Apparatus, method, and system for image-based human embryo cell classification

ABSTRACT

Apparatuses, methods, and systems for automated cell classification, embryo ranking, and/or embryo categorization are provided. An apparatus includes a classification module configured to apply classifiers to images of one or more cells to determine, for each image, a classification probability associated with each classifier. Each classifier is associated with a distinct first number of cells, and is configured to determine the classification probability for each image based on cell features including one or more machine learned cell features. The classification probability indicates an estimated likelihood that the distinct first number of cells is shown in each image. The classification module is further configured to classify each image as showing a second number of cells based on the distinct first number of cells and the classification probabilities associated therewith. The classification module is implemented in at least one of a memory or a processing device.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a Continuation of U.S. Ser. No. 14/194,386filed Feb. 28, 2014; which application claims priority to U.S.Provisional Appln. No. 61/785,170 filed Mar. 14, 2013; U.S. ProvisionalAppln. No. 61/785,216 filed Mar. 14, 2013; U.S. Provisional Appln. No.61/771,000 filed Feb. 28, 2013; U.S. Provisional Appln. No. 61/785,179filed Mar. 14, 2013; U.S. Provisional Appln. No. 61/785,199 filed Mar.14, 2013; and U.S. Provisional Appln. No. 61/770,998 filed Feb. 28,2013. The disclosures, all of which are incorporated herein by referencein their entirety for all purposes.

This application is related to copending U.S. patent application Ser.No. 14/194,391 filed Feb. 28, 2014. The content of which is incorporatedby reference in its entirety for all purposes.

FIELD OF THE INVENTION

The present invention relates generally to cell classification and/oroutcome determination. More particularly, this invention relates toclassification to determine characteristics associated with pluripotentcells such as, but not limited to, embryos, oocytes, and/or the like.Additionally or alternatively, this invention relates to embryo outcomedetermination for a series of embryo images with an unknown outcomebased on cell feature information extracted from one or more series ofembryo images with a known outcome. Additionally or alternatively, thisinvention relates to automated embryo ranking and/or categorization.

BACKGROUND OF THE INVENTION

Infertility is a common health problem that affects 10-15% of couples ofreproductive-age. In the United States alone in the year 2006,approximately 140,000 cycles of in vitro fertilization (IVF) wereperformed (cdc.gov/art). This resulted in the culture of more than amillion embryos annually with variable, and often ill-defined, potentialfor implantation and development to term. The live birth rate, percycle, following IVF was just 29%, while on average 30% of live birthsresulted in multiple gestations (cdc.gov/art). Multiple gestations havewell-documented adverse outcomes for both the mother and fetuses, suchas miscarriage, pre-term birth, and low birth rate. Potential causes forfailure of IVF are diverse; however, since the introduction of IVF in1978, one of the major challenges has been to identify the embryos thatare most suitable for transfer and most likely to result in termpregnancy.

Traditionally in IVF clinics, human embryo viability has been assessedby simple morphologic observations such as the presence ofuniformly-sized, mononucleate blastomeres and the degree of cellularfragmentation (Rijinders P M, Jansen C A M. (1998) Hum Reprod13:2869-73; Milki A A, et al. (2002) Fertil Steril 77:1191-5). Morerecently, additional methods such as extended culture of embryos (to theblastocyst stage at day 5) and analysis of chromosomal status viapreimplantation genetic diagnosis (PGD) have also been used to assessembryo quality (Milki A, et al. (2000) Fertil Steril 73:126-9; FragouliE, (2009) Fertil Steril June 21 [EPub ahead of print]; El-Toukhy T, etal. (2009) Reprod. Health 6:20; Vanneste E, et al. (2009) Nat Med15:577-83). However, potential risks of these methods also exist in thatthey prolong the culture period and disrupt embryo integrity(Manipalviratn S, et al. (2009) Fertil Steril 91:305-15; Mastenbroek S,et al. (2007) N Engl J. Med. 357:9-17).

Recently it has been shown that time-lapse imaging can be a useful toolto observe early embryo development and to correlate early developmentwith potential embryonic viability. Some methods have used time-lapseimaging to monitor human embryo development following intracytoplasmicsperm injection (ICSI) (Nagy et al. (1994) Human Reproduction.9(9):1743-1748; Payne et al. (1997) Human Reproduction. 12:532-541).Polar body extrusion and pro-nuclear formation were analyzed andcorrelated with good morphology on day 3. However, no parameters werecorrelated with blastocyst formation or pregnancy outcomes. Othermethods have looked at the onset of first cleavage as an indicator topredict the viability of human embryos (Fenwick, et al. (2002) HumanReproduction, 17:407-412; Lundin, et al. (2001) Human Reproduction16:2652-2657). However, these methods do not recognize the importance ofthe duration of cytokinesis or time intervals between early divisions.

Other methods have used time-lapse imaging to measure the timing andextent of cell divisions during early embryo development(WO2007/144001). However, these methods disclose only a basic andgeneral method for time-lapse imaging of bovine embryos, which aresubstantially different from human embryos in terms of developmentalpotential, morphological behavior, molecular and epigenetic programs,and timing and parameters surrounding transfer. For example, bovineembryos take substantially longer to implant compared to human embryos(30 days and 9 days, respectively). (Taft, (2008) Theriogenology69(1):10-16. Moreover, no specific imaging parameters or time intervalsare disclosed that might be predictive of human embryo viability.

While time-lapse imaging has shown promise in the context of automatedanalysis of early human embryo development, significant developmentand/or performance hurdles remain unaddressed by these preexistingmethods. The nature, timing, and other benchmarks of early human embryodevelopment provide challenges for predicting development behavior. Suchchallenges can include predicting and/or otherwise determining, viaimage processing, the number of cell divisions, the timing of celldivisions, and the health of the individual cells and/or zygote atvarious points during development. Specifically, automated tracking ofindividual cells, which forms the basis for each of thesedeterminations, can be difficult due to the inherently noisy nature ofbiological images, as may arise due to lack of distinct visual features,missing and/or false cell boundaries, changing topology of the cell massdue to the cell division/and or cell movement, cell shape deformation,and so on. Any further inference(s) from such automated tracking thencan inherit the tracking error(s).

For example, individual cell tracking errors can be furtherpropagated/magnified when the number of cells in each image obtained viaautomated tracking is the basis for estimating time(s) of cell divisionevent(s). As another example, when the estimated number of cells and/ordivision timing information is used to determine likelihood of futureembryo viability, this automated determination can also be erroneous,and can lead to erroneous decisions, such as whether to proceed with IVFusing particular embryos. In addition, when embryos are of similarquality, it can be difficult to differentiate the embryos to determine,for example, which embryos to implant and which embryos to freeze.

It is against this background that a need arose to develop theapparatuses, methods, and systems for image-based human embryo cellclassification, image-based embryo outcome determination, and forautomated embryo ranking and/or categorization described herein.

SUMMARY OF THE INVENTION

Apparatuses, methods, and systems for cell classification, outcomedetermination, automated embryo ranking, and/or automated embryocategorization are provided.

In one embodiment, a method for automated cell classification includesapplying a plurality of first classifiers to each of a plurality ofimages of one or more cells to determine, for each image, a firstclassification probability associated with each first classifier. Eachfirst classifier is associated with a distinct first number of cells.The classifier determines the first classification probability for theeach image based on a plurality of cell features including one or moremachine learned cell features. The first classification probabilityindicates a first estimated likelihood that the distinct first number ofcells associated with the each first classifier is shown in the eachimage. Each of the plurality of images thereby has a plurality of thefirst classification probabilities associated therewith. The methodfurther includes classifying each image as showing a second number ofcells based on the distinct first number of cells associated with theeach first classifier and the plurality of first classificationprobabilities associated therewith.

In one embodiment, an apparatus for automated cell classificationincludes a classification module. The classification module isconfigured to apply a plurality of first classifiers to each of aplurality of images of one or more cells to determine, for each image, afirst classification probability associated with each first classifier.Each first classifier is associated with a distinct first number ofcells, and is configured to determine the first classificationprobability for the each image based on a plurality of cell featuresincluding one or more machine learned cell features. The firstclassification probability indicates a first estimated likelihood thatthe distinct first number of cells associated with the each firstclassifier is shown in the each image. Each of the plurality of imagesthereby has a plurality of the first classification probabilitiesassociated therewith. The classification module is further configured toclassify each image as showing a second number of cells based on thedistinct first number of cells associated with the each first classifierand the plurality of first classification probabilities associatedtherewith. The classification module is implemented in at least one of amemory or a processing device.

In one embodiment, a system for automated cell classification includes acomputing apparatus configured for automated cell classification. Thecomputing apparatus is configured to apply a plurality of firstclassifiers to each of a plurality of images of one or more cells todetermine, for each image, a first classification probability associatedwith each first classifier. Each first classifier is associated with adistinct first number of cells, and is configured to determine the firstclassification probability for the each image based on a plurality ofcell features including one or more machine learned cell features. Thefirst classification probability indicates a first estimated likelihoodthat the distinct first number of cells associated with the each firstclassifier is shown in the each image. Each of the plurality of imagesthereby has a plurality of the first classification probabilitiesassociated therewith. The computing apparatus is further configured toclassify each image as showing a second number of cells based on thedistinct first number of cells associated with the each first classifierand the plurality of first classification probabilities associatedtherewith.

In one embodiment, a method for image-based outcome determinationincludes applying a classifier to a first time-sequential series ofimages of one or more cells to determine, for the first time-sequentialseries of images, a classification probability. The classificationprobability indicates an estimated likelihood that a first outcome fordevelopment of the one or more cells is shown by the firsttime-sequential series of images. The first outcome is included in aplurality of outcomes for cell development associated with theclassifier. The method further includes classifying the first time-lapseseries of images as showing the first outcome based on the plurality ofoutcomes associated with the classifier and the classificationprobability.

In one embodiment, an apparatus for image-based outcome determinationincludes a classification module. The classification module isconfigured to apply a classifier to a first time-sequential series ofimages of one or more cells to determine, for the first time-sequentialseries of images, a classification probability. The classificationprobability indicates an estimated likelihood that a first outcome fordevelopment of the one or more cells is shown by the firsttime-sequential series of images. The first outcome is included in aplurality of outcomes for cell development associated with theclassifier. The classification module is further configured to classifythe first time-lapse series of images as showing the first outcome basedon the plurality of outcomes associated with the classifier and theclassification probability. The classification module is implemented inat least one of a memory or a processing device.

In one embodiment, a system for image-based outcome determinationincludes a computing apparatus configured for image-based outcomedetermination. The computing apparatus is configured to apply aclassifier to a first time-sequential series of images of one or morecells to determine, for the first time-sequential series of images, aclassification probability. The classification probability indicates anestimated likelihood that a first outcome for development of the one ormore cells is shown by the first time-sequential series of images. Thefirst outcome is included in a plurality of outcomes for celldevelopment associated with the classifier. The computing apparatus isfurther configured to classify the first time-lapse series of images asshowing the first outcome based on the plurality of outcomes associatedwith the classifier and the classification probability.

In one embodiment, a method for automated embryo ranking includesapplying one or more of a classifier and a neural network to a pluralityof images of each embryo included in a first plurality of embryos todetermine a score associated with a developmental potential of eachembryo included in the first plurality of embryos. The method furtherincludes ranking each embryo included in the first plurality of embryosbased on the score associated with each embryo included in the firstplurality of embryos.

In one embodiment, an apparatus for automated embryo ranking includes ascore determination module configured to classify a plurality of imagesof each embryo included in a first plurality of embryos to determine ascore associated with a developmental potential of each embryo includedin the first plurality of embryos. The apparatus further includes aranking module configured to rank each embryo included in the firstplurality of embryos based on the score associated with each embryoincluded in the first plurality of embryos. The score determinationmodule and the ranking module are implemented in at least one of amemory or a processing device.

In one embodiment, a method for automated embryo categorization includeapplying one or more of a classifier and a neural network to a pluralityof images of each embryo included in a first plurality of embryos todetermine a score associated with a developmental potential of eachembryo included in the first plurality of embryos. The method furtherincludes associating each embryo included in the first plurality ofembryos with a corresponding category included in a plurality ofcategories based on the score associated with each embryo included inthe first plurality of embryos.

In one embodiment, an apparatus for automated embryo categorizationincludes a score determination module configured to classify a pluralityof images of each embryo included in a first plurality of embryos todetermine a score associated with a developmental potential of eachembryo included in the first plurality of embryos. The apparatus furtherincludes a categorization module configured to associate each embryoincluded in the first plurality of embryos with a corresponding categoryincluded in a plurality of categories based on the score associated witheach embryo included in the first plurality of embryos. The scoredetermination module and the categorization module are implemented in atleast one of a memory or a processing device.

Other aspects and embodiments of the invention are also contemplated.The foregoing summary and the following detailed description are notmeant to restrict the invention to any particular embodiment but aremerely meant to describe some embodiments of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the nature and objects of the invention,reference should be made to the following detailed description taken inconjunction with the accompanying drawings, in which:

FIG. 1A illustrates an exemplary image-based cell classificationapproach, in accordance with an embodiment of the invention;

FIG. 1B illustrates exemplary training images, in accordance with anembodiment of the invention;

FIG. 1C illustrates feature vectors for each of a plurality of images,in accordance with an embodiment of the invention;

FIG. 2 illustrates exemplary image-based cell classification results bythe image-based cell classification approach of FIG. 1A, in accordancewith an embodiment of the invention;

FIG. 3A illustrates an image-based cell classification approach usingthe level-1 image classifier of FIG. 1A, in accordance with anembodiment of the invention;

FIG. 3B illustrates an image-based cell classification approach usingthe level-1 and level-2 image classifiers of FIG. 1A and FIG. 3A, inaccordance with an embodiment of the invention;

FIG. 3C illustrates an image-based cell classification refining approachusing a Viterbi classifier applied to the output of the level-2 imageclassifier of FIG. 3B, in accordance with an embodiment of theinvention;

FIG. 4 illustrates an exemplary cell development outcome determinationapproach, in accordance with an embodiment of the invention;

FIG. 5 illustrates an exemplary approach for unsupervised learning, inaccordance with an embodiment of the invention;

FIG. 6 illustrates an exemplary approach for feature extraction, inaccordance with an embodiment of the invention;

FIG. 7 illustrates an exemplary approach for outcome determination, inaccordance with an embodiment of the invention;

FIG. 8 illustrates a system for automated image-based cellclassification, image-based cell development outcome determination,embryo ranking, and/or embryo categorization in accordance with anembodiment of the invention;

FIG. 9 illustrates the computing apparatus of FIG. 8, in accordance withan embodiment of the invention;

FIG. 10 illustrates a method for image-based cell development outcomedetermination, in accordance with an embodiment of the invention;

FIG. 11 illustrates a method for automated image-based cellclassification, in accordance with an embodiment of the invention;

FIG. 12 illustrates an exemplary approach for image-based cellclassification, in accordance with an embodiment of the invention;

FIGS. 13A and 13B illustrate a bag of features in accordance with anexample, showing (a) examples of dense and sparse occurrence histogramsgenerated from sparsely detected descriptors and densely sampleddescriptors with a learned codebook; and (b) four examples of clusters(appearance codewords) generated by k-means clustering;

FIG. 14 illustrates an example of temporal image similarity;

FIG. 15A illustrates exemplary results for precision rate of celldivision detection as a function of offset tolerance obtained from anexemplary 3-level classification method, in accordance with anembodiment of the invention;

FIG. 15B illustrates exemplary results for recall rate of cell divisiondetection as a function of offset tolerance obtained from an exemplary3-level classification method, in accordance with an embodiment of theinvention;

FIG. 16A illustrates a non-limiting example of an automated celltracking approach applied to images of cell development such as embryodevelopment, in accordance with an embodiment of the invention;

FIG. 16B illustrates an expanded view of cell boundary segments shown inFIG. 16A, in accordance with an embodiment of the invention;

FIG. 17A illustrates a non-limiting example of a cell trackingframework, in accordance with an embodiment of the invention;

FIG. 17B illustrates a non-limiting example of a cell trackingframework, in accordance with another embodiment of the invention;

FIG. 18A illustrates a method for obtaining cell boundary featureinformation, in accordance with an embodiment of the invention;

FIG. 18B illustrates a method for generating a mapping of arepresentation of cells to cell boundary feature information andrefining hypotheses each including an inferred characteristic of one ormore of the cells, in accordance with an embodiment of the invention;

FIG. 18C illustrates a method for selecting hypotheses from thehypotheses illustrated in FIG. 16A, in accordance with an embodiment ofthe invention;

FIG. 19A illustrates an exemplary approach for selection of hypotheses1612 for the images 1602 of FIG. 16A, in accordance with an embodimentof the invention;

FIG. 19B illustrates an exemplary approach for selection of hypotheses1612 for the images 1602 of FIG. 16A, in accordance with an embodimentof the invention;

FIG. 19C illustrates an exemplary and nonlimiting approach fordetermination of a confidence measure for selected hypotheses (such asselected hypotheses 1612 of FIG. 16A) and for applying this confidenceinformation, according to an embodiment of the invention;

FIG. 20 illustrates a method for automated evaluation of cell activity,in accordance with an embodiment of the invention;

FIG. 21 illustrates a method for automated evaluation of cell activityincluding reliability determination, in accordance with an embodiment ofthe invention;

FIG. 22 illustrates exemplary results for ratio of embryos for whichrmsd<d_(p)+m on (i) transitions t₁, (ii) transitions t₂), (iii)transitions t₃, and (iv) all 3, when using: (a) classifier andsimilarity measure (tracking free), (b) DD-SMC max marginals (trackingbased), and (c) all observables (combined), in accordance with anembodiment of the invention;

FIG. 23 illustrates a non-limiting example of an automated embryoranking and/or embryo categorization approach applied to images ofembryos, in accordance with an embodiment of the invention;

FIG. 24 illustrates a non-limiting example of a display showing a resultof automated embryo ranking and embryo categorization, in accordancewith an embodiment of the invention; and

FIG. 25 illustrates a method for embryo ranking and/or embryocategorization, in accordance with an embodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

Before the present apparatuses, systems, and methods are described, itis to be understood that this invention is not limited to the particularapparatus, system, or method described, as such may, of course, vary. Itis also to be understood that the terminology used herein is for thepurpose of describing particular embodiments only, and is not intendedto be limiting, since the scope of the present invention will be limitedonly by the appended claims.

Where a range of values is provided, it is understood that eachintervening value, to the tenth of the unit of the lower limit unlessthe context clearly dictates otherwise, between the upper and lowerlimits of that range is also specifically disclosed. Each smaller rangebetween any stated value or intervening value in a stated range and anyother stated or intervening value in that stated range is encompassedwithin the invention. The upper and lower limits of these smaller rangesmay independently be included or excluded in the range, and each rangewhere either, neither or both limits are included in the smaller rangesis also encompassed within the invention, subject to any specificallyexcluded limit in the stated range. Where the stated range includes oneor both of the limits, ranges excluding either or both of those includedlimits are also included in the invention.

Unless defined otherwise, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art to which this invention belongs. Although any methods andmaterials similar or equivalent to those described herein can be used inthe practice or testing of the present invention, some potential andpreferred methods and materials are now described. All publicationsmentioned herein are incorporated herein by reference to disclose anddescribe the methods and/or materials in connection with which thepublications are cited. It is understood that the present disclosuresupersedes any disclosure of an incorporated publication to the extentthere is a contradiction.

It must be noted that as used herein and in the appended claims, thesingular forms “a”, “an”, and “the” include plural referents unless thecontext clearly dictates otherwise. Thus, for example, reference to “acomputer” includes a plurality of such computers known to those skilledin the art, and so forth.

It must be noted that as used herein and in the appended claims, theterm “one or more of” with a listing of items, such as, for example,“one or more of A and B”, means A (without B), B (without A), or A andB. Thus, for example, “one or more of a classifier and a neural network”means a classifier, a neural network, or a classifier and a neuralnetwork.

Any publications discussed herein are provided solely for theirdisclosure prior to the filing date of the present application. Nothingherein is to be construed as an admission that the present invention isnot entitled to antedate such publication by virtue of prior invention.Further, the dates of publication provided may be different from theactual publication dates which may need to be independently confirmed.

DEFINITIONS

The terms “developmental potential” and “developmental competence” areused herein to refer to the ability or capacity of a healthy embryo orpluripotent cell to grow or develop.

The term “embryo” is used herein to refer both to the zygote that isformed when two haploid gametic cells, e.g., an unfertilized secondaryoocyte and a sperm cell, unite to form a diploid totipotent cell, e.g.,a fertilized ovum, and to the embryo that results from the immediatelysubsequent cell divisions, i.e. embryonic cleavage, up through themorula, i.e. 16-cell stage and the blastocyst stage (with differentiatedtrophoectoderm and inner cell mass).

The term “pluripotent cell” is used herein to mean any cell that has theability to differentiate into multiple types of cells in an organism.Examples of pluripotent cells include stem cells oocytes, and 1-cellembryos (i.e. zygotes).

The term “stem cell” is used herein to refer to a cell or a populationof cells which: (a) has the ability to self-renew, and (b) has thepotential to give rise to diverse differentiated cell types. Frequently,a stem cell has the potential to give rise to multiple lineages ofcells. As used herein, a stem cell may be a totipotent stem cell, e.g. afertilized oocyte, which gives rise to all of the embryonic andextraembryonic tissues of an organism; a pluripotent stem cell, e.g. anembryonic stem (ES) cell, embryonic germ (EG) cell, or an inducedpluripotent stem (iPS) cell, which gives rise to all of embryonictissues of an organism, i.e. endoderm, mesoderm, and ectoderm lineages;a multipotent stem cell, e.g. a mesenchymal stem cell, which gives riseto at least two of the embryonic tissues of an organism, i.e. at leasttwo of endoderm, mesoderm and ectoderm lineages, or it may be atissue-specific stem cell, which gives rise to multiple types ofdifferentiated cells of a particular tissue. Tissue-specific stem cellsinclude tissue-specific embryonic cells, which give rise to the cells ofa particular tissue, and somatic stem cells, which reside in adulttissues and can give rise to the cells of that tissue, e.g. neural stemcells, which give rise to all of the cells of the central nervoussystem, satellite cells, which give rise to skeletal muscle, andhematopoietic stem cells, which give rise to all of the cells of thehematopoietic system.

The term “oocyte” is used herein to refer to an unfertilized female germcell, or gamete. Oocytes of the subject application may be primaryoocytes, in which case they are positioned to go through or are goingthrough meiosis I, or secondary oocytes, in which case they arepositioned to go through or are going through meiosis II.

By “meiosis” it is meant the cell cycle events that result in theproduction of gametes. In the first meiotic cell cycle, or meiosis I, acell's chromosomes are duplicated and partitioned into two daughtercells. These daughter cells then divide in a second meiotic cell cycle,or meiosis II, that is not accompanied by DNA synthesis, resulting ingametes with a haploid number of chromosomes.

By a “mitotic cell cycle”, it is meant the events in a cell that resultin the duplication of a cell's chromosomes and the division of thosechromosomes and a cell's cytoplasmic matter into two daughter cells. Themitotic cell cycle is divided into two phases: interphase and mitosis.In interphase, the cell grows and replicates its DNA. In mitosis, thecell initiates and completes cell division, first partitioning itsnuclear material, and then dividing its cytoplasmic material and itspartitioned nuclear material (cytokinesis) into two separate cells.

By a “first mitotic cell cycle” or “cell cycle 1” it is meant the timeinterval from fertilization to the completion of the first cytokinesisevent, i.e. the division of the fertilized oocyte into two daughtercells. In instances in which oocytes are fertilized in vitro, the timeinterval between the injection of human chorionic gonadotropin (HCG)(usually administered prior to oocyte retrieval) to the completion ofthe first cytokinesis event may be used as a surrogate time interval.

By a “second mitotic cell cycle” or “cell cycle 2” it is meant thesecond cell cycle event observed in an embryo, the time interval betweenthe production of daughter cells from a fertilized oocyte by mitosis andthe production of a first set of granddaughter cells from one of thosedaughter cells (the “leading daughter cell”, or daughter cell A) bymitosis. Upon completion of cell cycle 2, the embryo consists of 3cells. In other words, cell cycle 2 can be visually identified as thetime between the embryo containing 2-cells and the embryo containing3-cells.

By a “third mitotic cell cycle” or “cell cycle 3” it is meant the thirdcell cycle event observed in an embryo, typically the time interval fromthe production of daughter cells from a fertilized oocyte by mitosis andthe production of a second set of granddaughter cells from the seconddaughter cell (the “lagging daughter cell” or daughter cell B) bymitosis. Upon completion of cell cycle 3, the embryo consists of 4cells. In other words, cell cycle 3 can be visually identified as thetime between the embryo containing 3-cells and the embryo containing4-cells.

By “first cleavage event”, it is meant the first division, i.e. thedivision of the oocyte into two daughter cells, i.e. cell cycle 1. Uponcompletion of the first cleavage event, the embryo consists of 2 cells.

By “second cleavage event”, it is meant the second set of divisions,i.e. the division of leading daughter cell into two granddaughter cellsand the division of the lagging daughter cell into two granddaughtercells. In other words, the second cleavage event consists of both cellcycle 2 and cell cycle 3. Upon completion of second cleavage, the embryoconsists of 4 cells.

By “third cleavage event”, it is meant the third set of divisions, i.e.the divisions of all of the granddaughter cells. Upon completion of thethird cleavage event, the embryo typically consists of 8 cells.

By “cytokinesis” or “cell division” it is meant that phase of mitosis inwhich a cell undergoes cell division. In other words, it is the stage ofmitosis in which a cell's partitioned nuclear material and itscytoplasmic material are divided to produce two daughter cells. Theperiod of cytokinesis is identifiable as the period, or window, of timebetween when a constriction of the cell membrane (a “cleavage furrow”)is first observed and the resolution of that constriction event, i.e.the generation of two daughter cells. The initiation of the cleavagefurrow may be visually identified as the point in which the curvature ofthe cell membrane changes from convex (rounded outward) to concave(curved inward with a dent or indentation). The onset of cell elongationmay also be used to mark the onset of cytokinesis, in which case theperiod of cytokinesis is defined as the period of time between the onsetof cell elongation and the resolution of the cell division.

By “first cytokinesis” or “cytokinesis 1” it is meant the first celldivision event after fertilization, i.e. the division of a fertilizedoocyte to produce two daughter cells. First cytokinesis usually occursabout one day after fertilization.

By “second cytokinesis” or “cytokinesis 2”, it is meant the second celldivision event observed in an embryo, i.e. the division of a daughtercell of the fertilized oocyte (the “leading daughter cell”, or daughterA) into a first set of two granddaughters.

By “third cytokinesis” or “cytokinesis 3”, it is meant the third celldivision event observed in an embryo, i.e. the division of the otherdaughter of the fertilized oocyte (the “lagging daughter cell”, ordaughter B) into a second set of two granddaughters.

After fertilization both gametes contribute one set of chromosomes,(haploid content), each contained in a structure referred to herein as a“pronucleus” (PN). After normal fertilization, each embryo shows two PN,one representing the paternal genetic material and one representing thematernal genetic material. “Syngamy” as used herein refers to thebreakdown of the PN when the two sets of chromosomes unite, occurringwithin a couple of hours before the first cytokinesis.

DESCRIPTION OF EMBODIMENTS OF THE INVENTION

Aspects of the invention are operable for image-based cellclassification and/or image based cell development outcome determinationusing one or more classifiers. In some embodiments, at least oneclassifier is usable for both cell classification and for outcomedetermination. In other embodiments, one or more classifiers usable forcell classification are different from one or more classifiers usablefor outcome determination.

In some embodiments, cell classification can include determining anumber of cells in the image. In some embodiments, cell classificationcan include determining a classification probability that the imagecontains a predicted number of cells; in some embodiments, the cellclassification can include a binary classification of the image ascontaining the predicted number of cells or not.

In some embodiments, one or more classifiers can each be applied to eachof a plurality of images of one or more cells. The plurality of imagescan be a time-sequential series of images, such as a time-lapse seriesof images. The cells shown in the plurality of images can be any cell ofinterest. In some embodiments, a number of the cells in each image is ofinterest, and can be determined by aspects of the invention. Forexample, the cells can be a human embryo, and the number of cells can berepresentative of the embryo at one or more of the one cell stage, thetwo cell stage, the three cell stage, the four cell stage, and so on. Insome embodiments, the four cell stage represents four or more cells.Other examples of such cells of interest include, but are not limitedto, oocytes and pluripotent cells.

Any suitable classifier may be employed. In some embodiments, theclassifier is based on a machine learning algorithm. The classifier maybe an AdaBoost (adaptive boosting) classifier, or another classifiersuch as a Support Vector Machine (SVM). In some embodiments, theclassifier is based on cell feature and/or pattern recognition. A cellfeature is a feature obtained based on one or more images of one or morecells (such as an embryo, oocyte, or pluripotent cell), such as, but notlimited to, recognition of cell shape, texture, edges, and/or the like.A cell feature is not limited to features associated with only a singlecell, and can refer to features associated with multiple cells and/orfeatures associated with one or more cells and another portion of animage showing the one or more cells, such as the image background. Insome embodiments, the classifier is trained via one or more supervisedlearning approaches, such as by using labeled images. In someembodiments, cell features on which the classifier is based aredetermined through one or more unsupervised learning approaches. Theseunsupervised learning approaches may use unlabeled images.

In some embodiments, a plurality of classifiers can be employed, eachassociated with a distinct number of cells. Further, in someembodiments, multiple levels of image classifiers can be employed, wherewithin each level, each classifier is associated with a distinct numberof cells. For the sake of clarity, an individual classifier associatedwith n number of cells will be identified as a cell classifier, and agrouping of classifiers (each applied to an image) will be referred toas an image classifier. In some embodiments, a refining algorithm can beapplied to the output of the last image classifier to further refine theclassification of the image. In some embodiments, the refining algorithmrefines the classification of each image based on a temporal imagesimilarity measure of the image. In some embodiments, the refiningalgorithm is a dynamic programming algorithm for finding the most likelyclassification of the images included in the time-lapse series ofimages. In some embodiments, the refining algorithm is a Viterbialgorithm.

In some embodiments, outcome determination can include determining apredicted outcome of several possible outcomes for a plurality of testimages of cell development with an unknown outcome. In some embodiments,outcome determination can include binary classification of the testimages, i.e. determining an outcome of two possible outcomes for thetest images.

In some embodiments, one or more classifiers can each be applied to aplurality of test images of one or more cells to perform outcomedetermination. The test images can be a time-sequential series ofimages, such as a time-lapse series of images. The series of images canbe included in a video of the one or more cells, such as a time-lapsevideo. The cells shown in the test images can be any cell of interest.For example, the cells can be a human embryo, and the possible outcomeof the test images can be either blast (i.e. a blastocyst is formed thatis suitable for implantation) or arrested (i.e. no blastocyst formationoccurs because the embryo development is arrested). Other examples ofsuch cells of interest include, but are not limited to, oocytes andpluripotent cells.

In some embodiments, the classifier is trained to perform outcomedetermination based on cell feature and/or pattern recognition, such as,but not limited to, recognition of cell shape, texture, edges, and/orthe like.

In some embodiments, cell features on which the classifier is based aredetermined through one or more unsupervised learning approaches. Theseunsupervised learning approaches can use unlabeled images. Accordingly,in some embodiments, the cell features can include one or more machinelearned cell features. Generally, the machine learned cell features canbe any cell feature that is learned from learning images, for thepurpose of subsequent use in outcome determination, cell classification,and/or the like. In some embodiments, the machine learned cell featurescan be based on unsupervised learning by the classifier from a pluralityof unlabeled learning images, the cell features being termed as a ‘bagof features’ in some embodiments. It is understood that the unlabeledlearning images may or may not form a time-lapse series. In someembodiments, the bag of features can be applied towards cellclassification by the classifier, as briefly described above anddescribed in more detail later.

In some embodiments the classifier, after unsupervised learning, istrained on at least one series of training images that is labeled and/orotherwise associated with a specified outcome, i.e. the classifierundergoes supervised training. In some embodiments, the classifier istrained on multiple series of training images, with at least one seriesof training images for each specified outcome provided. In someembodiments, the classifier is trained based on cell feature and/orpattern information extracted from each series of training imagesassociated with the respective specified outcome. In this manner, theclassifier can be trained to recognize cell feature informationassociated with each specified outcome, and can subsequently be appliedto classify the test images based on the specified outcome to which thecell feature information extracted from the test images bestcorresponds.

In some embodiments, the classifier can determine, for one or more cellsshown by the test images, a classification probability associated witheach specified outcome that indicates an estimated likelihood that thespecified outcome is shown by the test images. The classificationprobability can indicate an estimated likelihood of the specifiedoutcome for development of the one or more cells shown by the testimages. The classifier can then classify the test images based on theclassification probability such as by, for example, determining that thetest images show the specified outcome associated with the highestclassification probability.

FIG. 1A illustrates a non-limiting example of a 2-level image-based cellclassification approach that employs four AdaBoost cell classifiers ateach level, the four cell classifiers (i.e. the 1-cell classifier 102-1,the 2-cell classifier 102-2, the 3-cell classifier 102-3, and the 4-cell(or 4 or more cell) classifier 102-4) classifying an input image forshowing one cell, two cells, three cells, and four cells, respectively,in accordance with an embodiment of the invention. As illustrated inFIG. 1A, and as will be described in more detail later, the output ofthe level-1 image classifier 102 can be accounted for by the level-2image classifier 104 as additional features. In some embodiments, arefining algorithm such as a Viterbi algorithm, for example, is appliedto the output of the level-2 image classifier 104.

In some embodiments, each cell classifier of the image classifier candetermine, for each image, a first classification probability associatedwith each cell classifier. The first classification probability for theeach image can be based on a plurality of cell features. In someembodiments, the cell features can include one or more machine learnedcell features. Generally, the machine learned cell features can be anycell feature that is learned from learning images, for the purpose ofsubsequent use in cell classification. In some embodiments, the machinelearned cell features can be based on unsupervised learning by theclassifier on a plurality of unlabeled learning images having an unknownnumber of cells (also referred to as a ‘bag of features’). In someembodiments, the classifier learns the bag of features in this mannerfrom the plurality of unlabeled images, as described above for outcomedetermination, and as described in more detail later.

In one embodiment, the bag of features is based on keypoint descriptors,such as Scale-Invariant Feature Transform (SIFT), Speeded Up RobustFeatures (SURF), Fast Regina Keypoint (FREAK), and Binary RobustInvariant Scalable Keypoints (BRISK), or other suitable descriptorsknown to one of skill in the art.

In some embodiments, the cell features can also include one or morehand-crafted cell features, which are human-designed rather than machinelearned. Hand-crafted cell features may include region properties, GLCM,LBP, Hessian features, Gabor features, and/or cell boundary features(see Table 1).

Table 1 below illustrates an exemplary listing of six types ofhand-crafted features and the bag of features (determined throughunsupervised learning) that can be employed for per-imageclassification. The GLCM, LBP, and Gabor features are known texturefeatures that can be used for classification. Hessian features arestatistics computed from the first eigenvalues of Hessian-filteredimages that enhance cell edges. The region properties (area, number ofconvex hull points, solidity, eccentricity, and perimeter) can becomputed from a rough embryo mask obtained by applying a shortest pathalgorithm to extract the embryo boundary in polar image space. In otherembodiments, the features shown in Table 1 can be replaced withalternative feature sets and/or different numbers of features perfeature set. For example, the hand-crafted features in Table 1 can bereplaced with other machine learned features (such as features learnedbased on unsupervised learning) similar to the bag of features. Inanother example, a different number of features (such as 262 instead ofthe 369 shown in Table 1) can be used. In one embodiment, the 262features do not include the Boundary Features shown in Table 1, andinclude 200 features instead of 300 features in the Bag of Features.

TABLE 1 Cell features designed or learned automatically for per-imagecell classification Feature Set Number of Features Type Regionprops(area, solidity, 5 Shape eccentricity, etc.) GLCM (Gray-Level 22 TextureCo-occurrence Matrices) LBP (Local Binary 10 Texture Pattern Features)Hessian Features 15 Edge & Texture Gabor Features 10 Texture BoundaryFeatures 7 Edge (average angular score, continuity, etc.) Bag ofFeatures (features 300 Texture & learned from embyo images) Learned

FIG. 1B illustrates a non-limiting example of training images 108labeled as showing 1 cell (reference character 108-1), 2 cells(reference character 108-2), 3 cells (reference character 108-3), and 4cells (reference character 108-4), respectively, that can be used fortraining of the (for example) level-1 classifiers 102 of FIG. 1A, inaccordance with an embodiment of the invention. In one embodiment, thetraining images 108-4 may show 4 or more cells. The training of theclassifier may be supervised learning based on a plurality of labeledimages (such as the training images 108), each having a known number ofcells.

FIG. 1C illustrates an exemplary output 110 of a classifier employingthe listed features of Table 1 on a plurality of images, in accordancewith an embodiment of the invention. In FIG. 1C, each row 110-1 to 110-nis associated with a single image (also termed a ‘feature vector’ forthe image). In this embodiment, each feature vector 110-1 to 110-n has370 columns (not all shown in FIG. 1C), one for an image identifier 110a and one for each of the 369 features shown in Table 1. Representativecolumn entries 110 a-110 h, shown in FIG. 1C, include an imageidentifier 110 a and one representative entry (110 b-110 h) showing arepresentative value for a feature included in each of the seven featuresets listed in Table 1. These include representative entries 110 b-110 gassociated with the six hand crafted feature sets (Regionprops, GLCM,LBP, Hessian Features, Gabor Features, and Boundary Features), andrepresentative entry 110 h associated with the Bag of Features. In thismanner, each feature vector 110-1 to 110-n is representative of featureinformation in its associated image.

Once the cell classifier(s) have been trained, they can be applied tounlabeled images for per-image cell classification, as broadlyillustrated in the non-limiting example of FIG. 2, and as described inmore detail hereon. In FIG. 2, a time lapse series of images 202 of adeveloping embryo is classified on a per-image basis by the level-1 andlevel-2 classifiers 102, 104 of FIG. 1A, in accordance with anembodiment of the invention. The graph 204A, plotted as predicted numberof cells vs. image identifier associated with each of the plurality ofimages (e.g. such as a frame number or a time indicator), illustratesthe output of the level-1 classifiers 102 for each image of the images202, where the image identifier is provided on the X-axis, and theclassification result associated with the image identifier is providedon the Y-axis. The graph 204B illustrates the output of the level-2classifiers 104 for each image with the images 202 as well as the resultof level-1 classification as input. In general, unless noted otherwise,a plot of classification results or the result of applying a refiningalgorithm as disclosed herein is a plot of predicted number of cells vs.image identifier, where the image identifier is based on the series ofimages, and can be representative of the time each image was taken withrespect to each other image.

FIG. 3A illustrates per-image classification, in accordance with anembodiment of the invention. A series of test and/or otherwise unlabeledimages 302 serves as input to a level-1 image classifier 102 thatincludes 1-cell, 2-cell, 3-cell, and 4-cell (or 4 or more cell)classifiers (“cell classifier”, similar to the cell classifiers 102-1 to102-4 of FIG. 1 for example), and may further include additional cellclassifiers (not shown). In some embodiments, each classifier determinesa classification probability for each image based on cell features,where the cell features can be machine-learned features and/orhand-crafted cell features, as described earlier. In some embodiments,determining the classification probability includes extracting and/orotherwise determining a feature vector for each image (e.g. similar toeach row 110-1 to 110-n of FIG. 1C), and determining a classificationprobability based on the feature vector. Each classification probabilitycan be indicative of an estimated likelihood that the distinct number ofcells (e.g. 1 cell) associated with the each cell classifier (e.g. the1-cell classifier) is shown in the each image. In this manner, eachimages of the images 302 have a plurality of classificationprobabilities 304 associated therewith. In some embodiments, and asillustrated in FIG. 3A, the plurality of classification probabilities304 includes at least a 1-cell probability (reference character 304-1),a 2-cell probability (reference character 304-2), a 3-cell probability(reference character 304-3), and a 4-cell probability (referencecharacter 304-4), as suitably plotted in FIG. 3A for each cellclassifier. The output of the image classifier 102 can be represented bythe cumulative plot 306 of the output of all cell classifiers (“Level-1Output”). In general, unless noted otherwise, a plot of the output of animage classifier or cell classifier as disclosed herein is a plot ofclassification probability vs. image identifier, where the imageidentifier is based on the series of images, and can be representativeof the time each image was taken with respect to each other image.

Aspects of the invention are further configurable for classifying eachimage as showing a certain number of cells. In some embodiments, eachimage can be classified based on the distinct number of cells associatedwith each cell classifier and the plurality of classificationprobabilities associated therewith. For example, in FIG. 3A, each imageof the images 302 can be classified as showing 1 cell, 2 cells, 3 cells,or 4 cells (or 4 or more cells) based on the level-1 Output 306, whichprovides probability information for each cell number in each image. Theclassification of any image may be accomplished in any suitable mannerthat accounts for the classification probabilities associated with thatimage. In some embodiments, the image is deemed to be classified asshowing the cell number associated with the highest classificationprobability associated with that image. For example, the level-1 Output306 in FIG. 3A indicates that the highest classification probability forimage identifier 50 (e.g. representative of a time/timestampcorresponding to the 50^(th) image) of images 302 is from the 1-cellclassifier 102-1, and the highest classification probability for imageidentifier 450 of images 302 is from the 4-cell classifier 102-4.Accordingly, and as best illustrated in the “Level-1 Cell NumberClassification Result” plot 308 of FIG. 3A, image identifier 50 isclassified as showing 1 cell, while image identifier 450 is classifiedas showing 4 cells.

In some embodiments, the cell classification results 308 can be used toinfer biological activity based on one or more parameters such as cellactivity parameters, timing parameters, non-timing parameters, and/orthe like for the cells shown in the plurality of images. In someembodiments, the cell classification results 308 can be used to infercell division events based on the change(s) in the number of cells insuccessive images of a time-lapse series of images. For example, theclassification results 308 of FIG. 3A can be used to determine cellactivity parameters. In some embodiments, the parameters can includecell activity parameters, and be one or more of the following fordividing cells such as in a developing embryo: a duration of firstcytokinesis, a time interval between cytokinesis 1 and cytokinesis 2, atime interval between cytokinesis 2 and cytokinesis 3, a time intervalbetween a first and second mitosis, a time interval between a second andthird mitosis, a time interval from fertilization to an embryo havingfive cells (t5 in Table 2 below) and a time interval from syngamy to thefirst cytokinesis (S in Table 2 below).

In some embodiments, the parameters can include one or more parametersas described and/or referenced in Table 2 and/or other parameters,wherein the disclosures of (PCT Publication No.) WO 2012/163363, “EmbryoQuality Assessment Based on Blastomere Cleavage and Morphology,”International Filing Date May 31, 2012, (PCT Application No.)PCT/US2014/014449, “Abnormal Syngamy Phenotypes Observed With Time LapseImaging for Early Identification of Embryos With Lower DevelopmentPotential,” International Filing Date Feb. 3, 2014, and (PCT ApplicationNo.) PCT/US2014/014466, “Measuring Embryo Development and ImplantationPotential With Timing and First Cytokinesis Phenotype Parameters,”International Filing Date Feb. 3, 2014, are incorporated by reference intheir entireties.

TABLE 2 List of Parameters Parameter Description/Reference describingParameter P1 Duration of l^(st) cytokinesis P2 Interval between 1^(st)and 2^(nd) cytokinesis (time from 2-cell embryo to 3-cell embryo) (endof 1^(st) cytokinesis to end of 2^(nd) cytokinesis) (duration as 2 cellembryo) (t3-t2) P3 Interval between 2^(nd) and 3^(rd) cytokinesis (timefrom 3-cell embryo to 4-cell embryo) (end of 2^(nd) cytokinesis to endof 3^(rd) cytokinesis) (duration as 3 cell embryo) (t4-t3) (synchronybetween 3 and 4 cells) S Time from syngamy to 1^(st) cytokinesis 2ce-3CEnd of 1^(st) cleavage to beginning of second cleavage 3C-4C Beginningof 2^(nd) Cleavage to end of 3^(rd) Cleavage t5 Time from ICSI(fertilization) to 5 cell embryo 2Cb Time from fertilization tobeginning of 1^(st) cleavage 2Ce Time from fertilization until end of1^(st) cleavage 3C Time from fertilization to beginning of 2^(nd)cleavage 4C Time from fertilization to end of 3^(rd) cleavage 5C Timefrom fertilization to beginning of 4^(th) cleavage BL and/or ICSIFormation of blastocoel tM Time from fertilization to morula S3 Timefrom 5 cell embryo to 8 cell embryo t2 Time from fertilization to 2 cellembryo t3 Time from fertilization to 3 cell embryo t4 Time fromfertilization to 4 cell embryo cc3 T5-t3: Third cell cycle, duration ofperiod as 3 and 4 cell embryo t5-t2 Time to 5 cell embryo minus time to2 cell embryo cc3/cc2 Ratio of duration of cell cycle 3 to duration ofcell cycle 2 Time till first Duration of 1^(st) cell cycle cleavage 2PBExtrusion Time from fertilization until the second polar body isextruded PN fading Time from fertilization until pronuclei disappear, ORtime between the appearance of pronuclei appearing and pronucleidisappearing tSB Time from fertilization to the start of blastulationtSC Time from fertilization to the start of compaction PN appearanceTime from fertilization until pronuclei appear t6 Time fromfertilization to 6 cell embryo t7 Time from fertilization to 7 cellembryo t8 Time from fertilization to 8 cell embryo cc2b t4-t2; Secondcell cycle for both blastomeres, duration of period as 2 and 3 cellblastomere embryo cc2_3 t5-t2; Second and third cell cycle, duration ofperiod as 2, 3, and 4 blastomere embryo cc4 t9-t5; fourth cell cycle;duration of period as 5, 6, 7 and 8 blastomere embryo. s3a t6-t5;Duration of the individual cell divisions involved in the developmentfrom 4 blastomere embryo to 8 blastomere embryo s3b t7-t6; Duration ofthe individual cell divisions involved in the development from 4blastomere embryo to 8 blastomere embryo s3c t8-t7; Duration of theindividual cell divisions involved in the development from 4 blastomereembryo to 8 blastomere embryo cc2/cc3 WO 2012/163363 cc2/cc2_3 WO2012/163363 cc3/t5 WO 2012/163363 s2/cc2 WO 2012/163363 s3/cc3 WO2012/163363 AC1 Cleavage directly from 1 cell embryo to 3 cell embryoAC2 Cleavage of a daughter cell into more than 1 blastomere AS (abnormalsyngamy) Breakdown of pronuclei when two sets of chromosomes unite.Identified when PN disappear smoothly within the cytoplasm and normallyoccurs within a few hours prior to the first cytokinesis. MN2Multinucleation observed at 2 blastomere stage MN4 Multinucleationobserved at 4 blastomere stage EV2 Evenness of the blastomeres in the 2blastomere embryo Mul Multinucleation Uneven Uneven sizes of blastomeresat 2-4 cells Frg Fragmentation Nec Blastomere necrosis Vac Vacuolization

In some embodiments, one or more predictive criterion can be applied tothe one or more cells based on the determined cell activity parameters,such as, but not limited to, a measure of embryo quality (i.e. when theimages are of a developing embryo). In some embodiments, the predictivecriterion can be further employed to determine a predicted outcome suchas, for example, which embryo(s) will reach blastocyst, and can enablethe user to determine which embryo(s) have development potential forhuman implantation.

In some embodiments, the per-image probabilities 304-1 to 304-4 and/orclassification results 308 described above can be used to defineadditional cell features that can be used as input for anotherclassification process/approach. The exemplary embodiment of FIG. 3Billustrates how the classification probabilities 304-1 to 304-4determined by the cell classifiers of FIG. 3A (also referred to as“level-1 cell classifiers”, “first cell classifiers”, “level-1 imageclassifier” and/or “first image classifier”) can be employed tocalculate and/or otherwise determine additional cell features that canbe employed by a subsequent level-2 cell classifier 104 to classify theimages 302, in accordance with an embodiment of the invention. Asillustrated in FIG. 3B, the level-2 classifier 104 also includes 1-cell,2-cell, 3-cell and 4-cell classifiers that are associated with the1-cell, 2-cell, 3-cell and 4-cell classifiers of the level-1 classifier.In some embodiments, the additional cell features can be added to thefeature vector for each image to generate an enhanced feature vector. Asan example, an additional column can be added to the table 110illustrated in FIG. 1C for each additional cell feature, such that eachrow (associated with a single image) 110-1 to 110-n has an additionalentry for each additional cell feature. In some embodiments, one or moreadditional cell features are calculated for each image based on thelevel-1 classification probabilities 304-1 to 304-4 associated with thatimage, and based on the level-1 classification probabilities associatedwith at least one other image. For example, in some embodiments, four ormore additional cell features can be added to the feature vector 110-1to 110-n for each image based on the 1-cell, 2-cell, 3-cell and 4-cellclassification probabilities 304-1 to 304-4 respectively for that imageas determined by the level-1 cell classifiers. In some embodiments, thefour or more additional cell features added to the feature vector 110-1to 110-n for each image are based on one or more of an average (mean),median, maximum, minimum, standard deviation, and/or other combinedrepresentation of the 1-cell, 2-cell, 3-cell and 4-cell classificationprobabilities 304-1 to 304-4 respectively for that image and at leastone other image of the plurality of images. In some embodiments, theaveraged images are temporally adjacent to each other to facilitatereduction of noisy variations in the level-1 classificationprobabilities, such as those shown in the graph 204A of FIG. 2. In otherwords, with reference to the images 302, the averaged classificationprobabilities are adjacent to each other in the sequence of the images302. In this manner, classification information can be communicated fromone classifier to the next in a sequential image classification scheme.It is understood that while illustrated in FIG. 3B for two imageclassifiers 102 and 104, the approach(es) described herein areextendible to any additional image classifiers executing in sequenceand/or parallel. For example, in some embodiments, the output of thelevel-1 classifier can be fed in parallel to two or more level-2classifiers, each level-2 classifier having learned from and/or beingtrained on a different set of learning and/or training images,respectively. In this manner, aspects of the invention are operable toreceive independent, complementary validation of the output of eachlevel-2 classifier by comparing it to the output of each other level-2classifier.

Still referring to FIG. 3B, in some embodiments, each level-2 cellclassifier 104-1 to 104-4 is configured based on unsupervised learningon unlabeled learning images having an unknown number of cells. In someembodiments, the unlabeled images used for unsupervised learning of thelevel-2 cell classifiers 104-1 to 104-4 are different than at leastsome, if not all, the unlabeled images used for unsupervised learning ofthe level-1 cell classifier. Aspects of the invention are henceconfigurable for employing independently-trained classifiers in asequential manner such that each subsequent classification of an imagecan benefit from an independent prior classification of the same image.

Image-based cell classification (also referred to as “secondclassification”) by the level-2 image classifier 104 can proceed in amanner similar to that described above for the level-1 image classifier102. Namely, the level-2 cell classifiers 104-1 to 104-4 can be appliedto each image of images 302 to determine a second classificationprobability associated with each level-2 cell classifier for each image.In some embodiments, determining the second classification probabilitycan include extracting and/or otherwise determining an enhanced featurevector for each image as described above, and determining the secondclassification probability based on the feature vector. Each secondclassification probability can be indicative of an estimated likelihoodthat the distinct number of cells (e.g. 1 cell) associated with the eachcell classifier (e.g. the 1-cell classifier) is shown in the each image.In this manner, each image of the images 302 has a plurality of secondclassification probabilities associated therewith. In some embodiments,the plurality of second classification probabilities includes at least a1-cell probability, a 2-cell probability, a 3-cell probability, and a4-cell probability. The output of the level-2 image classifier can berepresented by the cumulative plot 310 of the output of all level-2 cellclassifiers (“Level-2 Output” plot).

Aspects of the invention are further configurable for classifying eachimage as showing a certain number of cells. In some embodiments, eachimage can be classified based on the distinct number of cells associatedwith each level-2 cell classifier and the second classificationprobabilities associated therewith. For example, in FIG. 3B, each imageof the images 302 can be classified as showing 1-cell, 2-cells, 3-cells,or 4-cells based on the level-2 Output, which provides secondprobability information for each cell number in each image. The secondclassification of any image may be accomplished in any suitable mannerthat accounts for the second classification probabilities associatedwith that image. In some embodiments, the image is deemed to beclassified as showing the cell number associated with the highest secondclassification probability associated with that image.

In some embodiments, the level-2 cell classification results 312 can beused to infer biological activity, cell activity parameters, and/or thelike for the cells shown in the plurality of images. In someembodiments, the level-2 cell classification results 312 can be used toinfer cell division events based on the change(s) in the number of cellsin successive images of a time-lapse series of images. For example, thelevel-2 classification results 312 of FIG. 3B can be used to determinecell activity parameters that include one or more of the following fordividing cells: a duration of first cytokinesis, a time interval betweencytokinesis 1 and cytokinesis 2, and a time interval between cytokinesis2 and cytokinesis 3, a time interval between a first and second mitosis,a time interval between a second and third mitosis, and a time intervalfrom fertilization to an embryo having five cells (t5 in Table 2).Alternatively or in addition, the level-2 classification results 312 ofFIG. 3B can be used to determine any of the cell activity parametersincluded in Table 2.

In some exemplary embodiments, and as illustrated in FIG. 3C, a Viterbialgorithm is used to refine the level-2 classification results 312 ofFIG. 3B. The level-2 classifier 104, or alternatively a module receivingthe level-2 classification results 312, may implement the Viterbialgorithm. The Viterbi algorithm can be used by the level-2 classifier104 to integrate prior knowledge, enforce the non-decreasing number ofcells, and fuse information such as classification probabilities andtemporal image similarity to generate final embryo stage classificationresults within a global context.

In some embodiments, for a given image, the Viterbi algorithm accountsfor each preceding image. The Viterbi algorithm may enforce thatsuccessive images have a non-decreasing number of cells, thereby‘smoothing’ the level-2 classification results, as illustrated in thelevel-3 results 314. In this manner, aspects of the invention canprovide a single most likely classification 314 of the images 302. Asalso shown in FIG. 3B, the Viterbi algorithm can also accept as input aTemporal Image Similarity Score 316 for each image, evaluated asdisclosed with reference to FIG. 14 below.

In some embodiments, one or more predictive criteria can be applied tothe one or more cells based on the determined cell activity parameters,such as, but not limited to, a measure of embryo quality (i.e. when theimages are of a developing embryo). In some embodiments, the predictivecriterion can be further employed to determine a hypothetical outcomesuch as, for example, which embryo(s) will reach blastocyst, and canenable the user to determine which embryo(s) have development potentialfor human implantation.

FIG. 4 illustrates a non-limiting example of an outcome determinationapproach for images of cell development such as embryo development,according to some embodiments of the invention. During training, N setsof training images 402 are provided with specified outcomes, eachspecified outcome here corresponding to either ‘blast’ or ‘arrested’.For example, as illustrated in FIG. 4, the series of training images402-1 is associated with the blast outcome, and the series of trainingimages 402-N is associated with the arrested outcome. In someembodiments, at least one or more of the training images 402 can be thesame as at least one or more of the training images 108. Alternatively,all of the training images 402 can be different from the training images108.

As also illustrated in FIG. 4, and as will be explained in more detailbelow, aspects of the invention are operable to carry out featureextraction 404 from each series of the training images 402. Theextracted feature information, such as one or more feature vectors, andtheir associated outcomes can be employed to train an outcome classifier406, although in some embodiments (not shown), multiple classifiers maybe trained on some or all of the series of training images 402. AlthoughFIG. 4 illustrates the classifier 406 as an AdaBoost classifier, it isunderstood that any suitable classifier may be employed.

The classifier 406, after training, can be applied for outcomedetermination of a series of test images 408. As illustrated in FIG. 4,feature information can be extracted from the test images 408 viafeature extraction 410 in a manner/approach similar to that used for thefeature extraction 404 for the training images 402. The classifier 406can then determine the outcome and/or classify the test images 408 asblast or arrested based on the extracted feature information. In someembodiments, and as also illustrated in FIG. 4, other related additionalor alternative outcomes/inferences may be determined by the classifier406, including whether the embryo is suitable for implantation or not(“Implantation, No-implantation”), and whether implantation of theembryo is likely to develop into a pregnancy or not (“Pregnancy,No-pregnancy”).

As discussed above, in some embodiments, the classifier undergoesunsupervised feature learning on unlabeled learning images to ‘learn’cell features, also called a bag of features. In one embodiment, the bagof features is based on keypoint descriptors, such as Scale-InvariantFeature Transform (SIFT), Speeded Up Robust Features (SURF), Fast ReginaKeypoint (FREAK), and Binary Robust Invariant Scalable Keypoints(BRISK), or other suitable descriptors known to one of skill in the art.

Any suitable learning approach may be employed that generates featureinformation representative of the learning images. In some embodiments,regions within each learning image are analyzed to determine a pluralityof local feature information associated with the regions of the learningimage (“local feature information”). In some embodiments, local featureinformation is determining by sampling the learning image at multiplelocations within the learning image. For example, the color, orintensity, at each sample point can be determined as a numeric value,and as the local feature information. In some embodiments, a compressedsensing technique, such as sparse sampling, is employed that accountsfor the sparse nature of information as is typical in biological images.In some embodiments, additional steps are taken towards detection and/ordescription of local features. For example, each sample point can befurther divided into bins, and multiple measurements can be made foreach bin for different directions to collect multiple local featuredescriptor values per sample.

In some embodiments, the local feature information can be combined toobtain image feature information for the entire learning image (“imagefeature information”). For example, the image feature information can bespecified as a multi-dimensional matrix, such as a two-dimensionalmatrix. The matrix may have a first dimension corresponding at least tothe number of sample points associated with the determination of thelocal feature information, and a second dimension corresponding at leastto additional detection/description information for each sample point,such as the number of local feature descriptor values collected persample. For example, in some embodiments, the feature descriptorsassociated with the learning image can be represented as thistwo-dimensional matrix, which can also be viewed as a collection offeature vectors associated with the local feature information. Thenumber of feature vectors may be the number of sample points, and thelength of each feature vector can be determined by the followingproduct: the number of bins×the number of directions, as described abovefor each sampling point.

In some embodiments, the image feature information for all learningimages can be combined to obtain feature information for the entireset/group of the learning images (“set feature information”). The setfeature information may include all of the local feature information forall of the learning images. For example, the set feature information canbe specified as a multi-dimensional matrix, such as a three-dimensionalmatrix. The matrix may have a first dimension corresponding at least tothe number of learning images, a second dimension corresponding at leastto the number of sample points associated with the determination of thelocal feature information, and a third dimension corresponding at leastto additional detection/description information for each sample point,such as the number of local feature descriptor values collected persample. In this manner, feature information at the local, image, and setlevel can be successively accounted for, aggregated, interlinked, and/orcombined in any suitable manner to generate set feature information fromwhich the outcome classifier can ultimately learns cell features.

In some embodiments, data mining approaches can be employed to divide upthe generated set feature information into a plurality of data regionsor clusters of relevant and/or useful cell feature information (“featureclusters”). In some embodiments, k-clustering approaches are employed,such as k-means clustering, k-median clustering, k-medoid clustering,and/or the like. In some embodiments, k-means clustering is employedthat partitions the set feature information into a plurality of featureclusters in which each observation belongs to the feature cluster withthe nearest mean. The set feature information can be represented by anysuitable number of feature clusters in this manner. In some embodiments,each feature cluster is representative of a learned cell feature, andcan be selected from feature types including but not limited to shapetype, edge type, and texture type. Any suitable representation of thefeature cluster can be employed, such as a plurality of visualizationsaround a centroid and/or other sampling point of the cluster. In someembodiments, a centroid or other sample point associated with each ofthe feature clusters (also known as a codeword) can be combined togenerate a codebook of the feature clusters, where the number of featureclusters may be the codebook size. For example, the codebook can bespecified as a multi-dimensional matrix, such as a two-dimensionalmatrix with matrix dimensions corresponding to the number of clustersand the number of local feature descriptor values per sample.

FIG. 5 illustrates an exemplary and non-limiting approach forunsupervised learning from unlabeled learning images, and shows anembodiment for determining each of the local feature information, theimage feature information, the set feature information, the plurality ofclusters, the codewords, and the codebook described above. It isunderstood that each of these may be determined by any suitable means,leading to a wide range of possibilities/combinations for generating thecodewords as the result of the unsupervised learning process.

FIG. 5 illustrates unsupervised learning in accordance with anembodiment of the invention, starting with a set of learning images 502.In some embodiments, at least one or more of the learning images 502 arethe same as at least one or more of the training images 108.Alternatively, all of the training images 502 can be different from thetraining images 108.

Each image 504 included in the learning images 502 is sampled togenerate local feature information, and accordingly, to generate imagefeature information for the image 504. The image feature information foreach image 504 included in the learning images 502 is represented by thematrix 506. As described earlier, the matrix 506 may have a firstdimension corresponding at least to the number of sample pointsassociated with the determination of the local feature information, anda second dimension corresponding at least to additionaldetection/description information for each sample point, such as thenumber of local feature descriptor values collected per sample. The setfeature information for the set of learning images 502 may berepresented as multiple matrices 506 (one representative matrix 506shown in FIG. 5), one per learning image 502, and/or as a singlethree-dimensional matrix incorporating the multiple matrices 506.

At 508, K-means clustering is applied to the matrices 506 (the setfeature information) to generate (in this example) 300 feature clustersin a 128-dimension feature space, each representing a learned cellfeature. At 510, each feature cluster is visualized around a samplingpoint, illustrated here as a 10×10 matrix of images for each featurecluster. In this example, the centroid of each feature cluster is acodeword, and a codebook 512 can then be generated based on the 300codewords, i.e. the codebook is of size 300 (number ofclusters/codewords)×128 (number of dimensions in feature space). Thecodebook 512 can serve as input to the outcome classifier as describing300 learned cell features for feature extraction from training and/ortest images.

Returning to FIG. 4, in some embodiments, upon unsupervised learning,the classifier 406 can be trained on the training images 402-1 to 402-Nfor the plurality of outcomes associated therewith. In some embodiments,training the classifier includes extracting local feature informationfrom each set of training images at 404, and associating the extractedfeature information with the outcome associated with the each set oftraining images by comparing the extracted local feature informationwith the learned codebook to generate series feature information. Inthis manner, the classifier 406 can be ‘trained’ to recognize a specificoutcome, and/or to determine the probability for a specific outcome, forthe test images 408 based on prior knowledge of what feature informationlooks like for that specific outcome.

In some embodiments, the feature extraction 404 for extracting trainingfeature information can operate as illustrated in the exemplary andnon-limiting approach of FIG. 6. A time-sequential series of trainingimages 602 (e.g. such as the training images 402-1) having a specifiedoutcome can be analyzed by the classifier 406 and/or a learning module940 (see FIG. 9) that may be included in or separate from the classifier406 to generate image feature information for each image 604 of theseries of training images. In some embodiments, at least one or more ofthe training images 602 can be the same as at least one or more of thetraining images 108. Alternatively, all of the training images 602 canbe different from the training images 108. In some embodiments, localfeature information for each image 604 is generated in a manner similarto that described above for unsupervised learning.

Referring to FIGS. 4-6, with the local feature information from thetraining images 302 and the codebook as input, the classifier 406 and/orthe learning module 940 (see FIG. 9) can then determine the frequencywith which each codeword occurs in the local feature information of eachtraining image 604 (which can be considered image feature information),and further determine the occurrence frequency for each codeword acrossall the training images 602, such as by averaging and/or determining themedian of the occurrence frequencies for each codeword in each of thetraining images 602. In this manner, the frequency distribution ofcodewords across all the training images 602, also termed the seriesfeature information, can be associated with the specified outcomeassociated with all the training images 602. As best illustrated in FIG.6, a histogram 606 (image feature information) visually depicts theresult of comparing the local feature information in training image 604against the codebook 512 generated in FIG. 5, and is a frequencydistribution of the frequency of occurrence of each of the 300 codewordsof codebook 512 in the image 604. A histogram 608 (series featureinformation) on the other hand, visually depicts the result of a)generating the frequency distribution data for each image of thetraining images 602, and b) averaging and/or determining the median ofthe frequency of occurrence of each codebook across all the images ofthe training images 602 to generate a single element that represents thefrequency of occurrence for each codeword in the codebook 512. Since thetraining images 602 can be a time-lapse series of images, the histogram608 can accordingly be representative of time-lapse information.Further, since the training images 602 are associated with the specifiedoutcome, the histogram 608 can accordingly be representative of thespecified outcome, and the classifier 406 can be considered ‘trained’ torecognize the specified outcome (e.g. the blast outcome, for thetraining images 402-1). By repeating this process with a different setof training images (e.g. with 402-N) having a different outcome (e.g.arrested outcome), the classifier can be considered trained todistinguish between the two outcomes. The classifier can now classify aseries of unlabeled images (such as the test images 108) based on thecodeword frequency distribution of the series of unlabeled images.

Once the classifier 406 has been trained on the set of possible outcomesfor outcome determination for the series of test images 408, theclassifier can be applied to the test images 408 of unknown outcome.Outcome determination can include feature extraction 410 of test local,image, and series feature information from the test images 408. In someembodiments, feature extraction 410 for the images 408 is carried out ina manner similar to the feature extraction 404, as illustrated in FIG. 6for each training image 604, and as described earlier. In other words,test local feature information is determined for each test image, whichcan be used to generate the test image feature information (i.e.codeword frequency distribution for each test image) for the each testimage, which in turn can be used to generate the test series featureinformation (i.e. combined codeword frequency distribution for theentire series) for the series of test images 408. An average testhistogram can be generated by applying the codebook to the local featureinformation in each of the test images, and by averaging and/ordetermining the median of the codeword frequency distribution soobtained.

With the histogram (“test histogram”) for the series of test images(e.g. the test images 408), and the average histogram 608 (“traininghistogram”) for each series of training images 402-1 to 402-N, theclassifier 406 can then determining a classification probability foreach outcome by performing classification of the series of test imagesbased on the test histogram and the training histogram(s) for thatspecified outcome. The classification can be performed in any suitablemanner, such as by an AdaBoost (adaptive boosting) classifier, oranother classifier such as a Support Vector Machine (SVM). Theclassifier 106 can then classify the test images as showing a predictedoutcome based on the classification probabilities associated with eachoutcome.

FIG. 7 illustrates an exemplary and non-limiting approach for outcomedetermination of the outcome 112 of FIG. 4. Training histograms 702-1 to702-N represent codeword frequency distributions for the correspondingseries of the training images 402-1 to 402-N, respectively, and furtherrepresent the specific outcome-1 to specific outcome-N, respectivelyassociated with the corresponding series of the training images 402-1 to402-N. Each training histogram 702 is compared against a test histogram706 that represents the codeword frequency distribution for the testimages 408 of FIG. 4, and classification probabilities 710-1 to 710-Nare determined that correspond to the specific outcome-1 to specificoutcome-N respectively. The outcome 412 is then determined based on theclassification probabilities 710-1 to 710-N.

In some embodiments, such as when the cells in the training/test imagesis an embryo, one or more predictive criterion can be applied based onthe determined outcome 412, such as, but not limited to, whether theembryo is suitable for implantation or not, whether the embryo, ifimplanted, will result in a pregnancy or not, and so on.

Referring to FIG. 8, a schematic diagram of a system 800 for automatedimage-based cell classification according to embodiments of theinvention is described. In other embodiments, the system 800 may be forimage-based embryo outcome determination. The system 800 includes atleast one imaging device 802, a computing apparatus 804, a displaydevice 806, and an input interface 808.

The imaging device 802 can be any device configurable to acquire imagesof one or more cells, such as the images 302, the training images forthe level-1 image classifier, the training images for the level-2 imageclassifier, and/or the like. The imaging device 502 can also beconfigurable to acquire a first time-sequential series of images such asthe test images 408 and to acquire a plurality of time-lapse series ofimages of one or more cells, such as the training images 402-1 to 402-N.The computing apparatus 804 can be configured to receive the images fromthe imaging device 802. In some embodiments, the imaging device 802includes a darkfield illumination microscope and/or a brightfieldillumination microscope, but is not limited to these imaging modalities.The display device 806 can be any suitable device for displaying controlinformation and/or data to a user of the system 800 (e.g. such as a LCDdisplay), and may optionally be suited for receiving user input (e.g.such as via a touch screen panel). In some embodiments, the displaydevice is at least configured to display one or more images of cells asacquired by the imaging device 802, and for presenting a characteristicof the cells based on the image classification described herein. In someembodiments, the display device 806 is at least configured present oneor more characteristics of the one or more cells in the first time-lapseseries of images based on one or more of the following: theclassification probability, the classifying, and the first outcome. Insome embodiments, the display device 806 is further configured topresent one or more characteristics of one or more cells in theplurality of time-lapse series of images based on the featureinformation.

In some embodiments, the computing apparatus 804, the display device806, and the input interface 808 may be integrated into a common chassis(such as in a personal computer, laptop, and/or tablet form factor), andmay be connected to the imaging device 802 over a wireline and/orwireless network. Alternatively or in addition, the imaging device 802,the computing apparatus 804, the display device 806, and the inputinterface 808 may be integrated into a common chassis.

In some embodiments, the computing apparatus 804 can be configured forimage-based outcome determination. In some embodiments, the computingapparatus 804 applies a classifier to a first time-sequential series ofimages of one or more cells to determine, for the first time-sequentialseries of images, a classification probability. In some embodiments, thefirst time-sequential series of images is a time-lapse series of images.In some embodiments, the classifier is an AdaBoost classifier. In someembodiments, the one or more cells is selected from the group consistingof: a human embryo, one or more oocytes, and one or more pluripotentcells.

In some embodiments, the classification probability indicates anestimated likelihood that a first outcome for development of the one ormore cells is shown by the first time-sequential series of images. Thefirst outcome can be included in a plurality of outcomes for celldevelopment associated with the classifier. The computing apparatus 804is further configured to classify the first time-lapse series of imagesas showing the first outcome based on the plurality of outcomesassociated with the classifier and the classification probability. Insome embodiments, the plurality of outcomes include one or more of thefollowing pairs of outcomes: blast and arrested; implantation and noimplantation; and pregnancy and no pregnancy.

In some embodiments, the computing apparatus 804 can be configured toconfigure each of a plurality of first classifiers based on a firstplurality of training images showing the distinct first number of cellsassociated with the each first classifier. In some embodiments, thecomputing apparatus 804 can be further configured to can be configuredto apply a plurality of first classifiers to each of a plurality ofimages of one or more cells to determine, for each image, a firstclassification probability associated with each first classifier. Theplurality of cell features can include one or more hand-crafted cellfeatures. In some embodiments, each of the plurality of cell featurescan be one or more of the following types: shape type, texture type, andedge type. In some embodiments, the plurality of first classifiers areAdaBoost classifiers configured to perform binary classification.

In some embodiments, each first classifier is associated with a distinctfirst number of cells, and the computing apparatus 804 can be configuredto determine the first classification probability for the each imagebased on a plurality of cell features including one or more machinelearned cell features.

In some embodiments, the first classification probability indicates afirst estimated likelihood that the distinct first number of cellsassociated with the each first classifier is shown in the each image.Each of the plurality of images thereby has a plurality of the firstclassification probabilities associated therewith.

In some embodiments, the computing apparatus 804 can be furtherconfigured to classify each image as showing a second number of cellsbased on the distinct first number of cells associated with the eachfirst classifier and the plurality of first classification probabilitiesassociated therewith.

In some embodiments, the computing apparatus 804 can be furtherconfigured to apply a plurality of second classifiers to each image todetermine, for the each image, a second classification probabilityassociated with each second classifier based on at least one of theplurality of the first classification probabilities. In someembodiments, at least one of the plurality of the first classificationprobabilities is associated with one or more of the plurality of imagesthat are temporally adjacent to the each image. In some embodiments, theplurality of images are a time-lapse series of images. In someembodiments, the second classification probability and the at least oneof the first classification probabilities are associated with the samedistinct first number of cells. In some embodiments, the plurality ofimages are a time-lapse series of images.

In some embodiments, the computing apparatus 804 can be configured toconfigure the plurality of second classifiers based on a secondplurality of training images showing the distinct third number of cellsassociated with the each second classifier. In some embodiments, each ofthe second plurality of training images is distinct from all of thefirst plurality of training images.

In some embodiments, the computing apparatus 804 can be furtherconfigured to apply the plurality of second classifiers to each image todetermine, for the each image, the second classification probabilityassociated with each second classifier. In some embodiments, each secondclassifier is associated with a distinct third number of cells, and theeach second classifier determines the second classification probabilityfor the each image based on the plurality of cell features, and furtherbased on one or more additional cell features associated with one ormore of the plurality of the first classification probabilitiesassociated with one or more images included in the plurality of imagesthat are temporally adjacent to the each image. In some embodiments, thesecond classification probability indicates a second estimatedlikelihood that the distinct third number of cells associated with theeach second classifier is shown in the each image. Each of the pluralityof images thereby has a plurality of the second classificationprobabilities associated therewith. In some embodiments, the distinctthird number of cells associated with the each second classifier isselected from the group consisting of one cell, two cells, three cells,and four or more cells. In some embodiments, the distinct third numberof cells associated with the each second classifier is the same as thedistinct first number of cells associated with a corresponding one ofthe plurality of first classifiers.

In some embodiments, the computing apparatus 804 can be furtherconfigured to classify each image as showing a fourth number of cellsbased on the distinct third number of cells associated with the eachsecond classifier and the plurality of second classificationprobabilities associated therewith. In some embodiments, the computingapparatus 804 can be further configured to apply a refining algorithm tothe plurality of images to determine, based on the plurality of images,that one or more of the plurality of images classified as showing thefourth number of cells instead shows a fifth number of cells differentfrom the fourth number of cells.

In some embodiments the computing apparatus 804 is further configured todetermine cell activity parameters of the one or more cells based on thefourth number of cells in the each image. In some embodiments, thedetermined cell activity parameters include one or more of thefollowing: a duration of first cytokinesis, a time interval betweencytokinesis 1 and cytokinesis 2, and a time interval between cytokinesis2 and cytokinesis 3, a time interval between a first and second mitosis,a time interval between a second and third mitosis, a time intervalbetween fertilization to an embryo having five cells and a time intervalbetween syngamy and the first cytokinesis.

In some embodiments, the computing apparatus 804 is further configuredto extract series feature information from the first time-sequentialseries of images and to apply the classifier to the firsttime-sequential series of images is based on the series featureinformation. In some embodiments, the series feature information isrepresentative of the first outcome and is associated with an entiretyof the first time-sequential series of images. In some embodiments, thecomputing apparatus 804 is further configured to extract the seriesfeature information by extracting local feature information associatedwith a portion of one or more of the first time-sequential series ofimages, and determining the series feature information based on thelocal feature information and a plurality of codewords.

In some embodiments, the computing apparatus 804 is further configuredto determine the series feature information by associating the localfeature information with one or more clusters, each of the one or moreclusters being associated with a corresponding one of the plurality ofcodewords. The computing apparatus 804 is further configured todetermine a frequency of occurrence of the one or more codewords acrossthe first time-sequential series of images, where the series featureinformation includes the frequency of occurrence of each of the one ormore codewords across the first time-sequential series of images. Insome embodiments, each of the plurality of codewords is associated witha cell feature that is one or more of the following: edge type, texturetype, and shape type.

In some embodiments, the computing apparatus 804 is further configuredto determine each of the plurality of codewords from a plurality ofunlabeled images of at least one cell through unsupervised learning.

In some embodiments, the computing apparatus 804 is further configuredto train the classifier based on series feature information associatedwith each of a plurality of time-sequential series of images, where theeach of the plurality of time-sequential series of images beingassociated with one of the plurality of outcomes. In some embodiments,the computing apparatus 804 is further configured to train theclassifier by extracting the series feature information from the each ofthe plurality of time-sequential series of images. In some embodiments,the series feature information associated with one of the plurality oftime-sequential series of images is representative of an associated oneof the plurality of outcomes, and is associated with an entirety of theone of the plurality of time sequential series of images.

In some embodiments, the computing apparatus 804 is further configuredto extract the series feature information by extracting local featureinformation associated with a portion of one or more of the plurality oftime-sequential series of images, and determine the series featureinformation based on the local feature information and a plurality ofcodewords determined from a plurality of unlabeled images of at leastone cell through unsupervised learning. In some embodiments, thecomputing apparatus 804 is further configured to determine the seriesfeature information by associating the local feature information withone or more clusters, where each of the one or more clusters beingassociated with a corresponding one of the plurality of codewords. Thecomputing apparatus 804 is further configured to determine a frequencyof occurrence of the one or more codewords across each of the one ormore of the plurality of time-sequential series of images. The seriesfeature information includes the frequency of occurrence of each of theone or more codewords across the each of the one or more of theplurality of time-sequential series of images. In some embodiments eachof the plurality of codewords is associated with a cell feature that isone or more of the following: edge type, texture type, and shape type.In some embodiments, the computing apparatus 504 is further configuredto determine each of the plurality of codewords from a plurality ofunlabeled images of at least one cell through unsupervised learning.

FIG. 9 illustrates the computing apparatus 804, in accordance with anembodiment of the invention. The computing apparatus 804 includes atleast a processor 912, a memory 914, an input/output module (I/O) 916,and connection interfaces 918 connected by a bus (not shown). In someembodiments, the memory 914 stores a set of executable programs (notshown) that are used to implement the computing apparatus 804 forautomated cell classification. Additionally or alternatively, theprocessor 912 can be used to implement the computing apparatus 804 forautomated cell classification, as illustrated in FIG. 8. The processor912 may include various combinations of the modules shown in FIG. 9,such as image module 920, a training module 934, a classification module936, an outcome determination module 938, a learning module 940, and adisplay module 942.

The image module 920 can be configured to receive a plurality of imagesof one or more cells. The image module 920 can be configured to acquirea first time-sequential series of images such as the test images 408 andto acquire a plurality of time-sequential series of images of one ormore cells, such as the training images 402-1 to 402-N. In someembodiments, the image module 520 also acquires the learning images.

The classification module 936 can be configured to apply a plurality offirst classifiers to each of the plurality of images of one or morecells to determine, for each image, a first classification probabilityassociated with each first classifier. Each first classifier can beassociated with a distinct first number of cells. The classificationmodule 936 can be further configured to determine the firstclassification probability for the each image based on a plurality ofcell features including one or more machine learned cell features. Thefirst classification probability can indicate a first estimatedlikelihood that the distinct first number of cells associated with theeach first classifier is shown in the each image. Each of the pluralityof images thereby has a plurality of the first classificationprobabilities associated therewith.

The classification module 936 can be further configured to classify eachimage as showing a second number of cells based on the distinct firstnumber of cells associated with the each first classifier and theplurality of first classification probabilities associated therewith.Each second classifier can be associated with a distinct third number ofcells. Each second classifier can determine the second classificationprobability for the each image based on the plurality of cell features,and further based on one or more additional cell features associatedwith one or more of the plurality of the first classificationprobabilities associated with one or more images included in theplurality of images that are temporally adjacent to the each image. Thesecond classification probability can indicate a second estimatedlikelihood that the distinct third number of cells associated with theeach second classifier is shown in the each image. Each of the pluralityof images thereby has a plurality of the second classificationprobabilities associated therewith. The classification module 936 can befurther configured to classify each image as showing a fourth number ofcells based on the distinct third number of cells associated with theeach second classifier and the plurality of second classificationprobabilities associated therewith.

The classification module 936 can be further configured to apply arefining algorithm to the plurality of images to determine, based on theplurality of images, that one or more of the plurality of imagesclassified as showing the fourth number of cells instead shows a fifthnumber of cells different from the fourth number of cells.

The classification module 936 can be configured to apply a classifier toa first time-sequential series of images of one or more cells todetermine, for the first time-sequential series of images, aclassification probability. The classification probability indicates anestimated likelihood that a first outcome for development of the one ormore cells is shown by the first time-sequential series of images. Thefirst outcome is included in a plurality of outcomes for celldevelopment associated with the classifier. The classification module936 can be further configured to classify the first time-lapse series ofimages as showing the first outcome based on the plurality of outcomesassociated with the classifier and the classification probability. Theclassification module 936 can be implemented on the processor 912 asshown. In addition or alternatively, the classification module 936 canbe implemented on the memory 914.

The training module 934 can be configured to configure each of theplurality of first classifiers based on a first plurality of trainingimages showing a distinct first number of cells associated with the eachfirst classifier. In some embodiments, the training module 934 can befurther configured to configure a plurality of second classifiers basedon a second plurality of training images showing a distinct third numberof cells associated with the each second classifier.

In some embodiments, the training module 934 is configured to extractseries feature information from the first time-sequential series ofimages, wherein the classification module 936 is further configured toapply the classifier to the first time-sequential series of images isbased on the series feature information. In some embodiments, thetraining module 934 is further configured to determine the seriesfeature information by associating the local feature information withone or more clusters, each of the one or more clusters being associatedwith a corresponding one of the plurality of codewords, and the learningmodule 940 is configured to determine each of the plurality of codewordsfrom a plurality of unlabeled images of at least one cell throughunsupervised learning.

In some embodiments, the processor 912 can further include a boundarydetection module 922, a hypothesis generation module 924, a hypothesisselection module 926, a mapping module 932, a cell activity parameterdetermination module 933, and an image similarity determination module941 for tracking of cell activity, as described in more detail inExamples 2 and 3 below.

In some embodiments, the processor 912 can further include a confidencemodule 928 and a reliability determination module 930 for human embryoviability screening, as described in more detail in Examples 2 and 3below.

In some embodiments, the processor can further include a selectionmodule 944, a score determination module 948, a ranking module 950, anda categorization module 952 for automated embryo ranking and/orcategorization, as described further below.

FIG. 10 illustrates a method 1000 of automated image-based cellclassification, in accordance with an embodiment of the invention. Insome embodiments, at least part of the method 1000 can be performed bythe computing apparatus 804, and by the classification module 932 inparticular. At step 1010, a plurality of first classifiers are appliedto each of a plurality of images of one or more cells to determine, foreach image, a first classification probability associated with eachfirst classifier. Each first classifier is associated with a distinctfirst number of cells, and determine the first classificationprobability for the each image based on a plurality of cell featuresincluding one or more machine learned cell features. The firstclassification probability can indicate a first estimated likelihoodthat the distinct first number of cells associated with the each firstclassifier is shown in the each image. Each of the plurality of imagesthereby has a plurality of the first classification probabilitiesassociated therewith.

At step 1020, each image is classified as showing a second number ofcells based on the distinct first number of cells associated with theeach first classifier and the plurality of first classificationprobabilities associated therewith.

At step 1030, a plurality of second classifiers are applied to eachimage to determine, for the each image, a second classificationprobability associated with each second classifier. Each secondclassifier is associated with a distinct third number of cells anddetermines the second classification probability for the each imagebased on the plurality of cell features, and further based on one ormore additional cell features associated with one or more of theplurality of the first classification probabilities associated with oneor more images included in the plurality of images that are temporallyadjacent to the each image. The second classification probabilityindicates a second estimated likelihood that the distinct third numberof cells associated with the each second classifier is shown in the eachimage, the each of the plurality of images thereby having a plurality ofthe second classification probabilities associated therewith.

At step 1040, each image is classified as showing a fourth number ofcells based on the distinct third number of cells associated with theeach second classifier and the plurality of second classificationprobabilities associated therewith.

In some embodiments, a method for automated cell classificationcomprises applying a plurality of first classifiers to each of aplurality of images of one or more cells to determine, for each image, afirst classification probability associated with each first classifier.Each first classifier is associated with a distinct first number ofcells, and determines the first classification probability for the eachimage based on a plurality of cell features including one or moremachine learned cell features. The first classification probabilityindicates a first estimated likelihood that the distinct first number ofcells associated with the each first classifier is shown in the eachimage, the each of the plurality of images thereby having a plurality ofthe first classification probabilities associated therewith.

In some embodiments, the method for automated cell classificationfurther includes classifying each image as showing a second number ofcells based on the distinct first number of cells associated with theeach first classifier and the plurality of first classificationprobabilities associated therewith.

In some embodiments, the distinct first number of cells associated withthe each first classifier is selected from the group consisting of onecell, two cells, three cells, and four or more cells

In some embodiments, each of the plurality of first classifiers isconfigured based on a first plurality of training images showing thedistinct first number of cells associated with the each firstclassifier.

In some embodiments, the plurality of cell features includes one or morehand-crafted cell features.

In some embodiments, the method for automated cell classificationfurther includes applying a plurality of second classifiers to eachimage to determine, for the each image, a second classificationprobability associated with each second classifier based on at least oneof the plurality of the first classification probabilities.

In some embodiments, the at least one of the plurality of the firstclassification probabilities is associated with one or more of theplurality of images that are temporally adjacent to the each image.

In some embodiments, the plurality of images are a time-lapse series ofimages.

In some embodiments, the second classification probability and the atleast one of the first classification probabilities are associated withthe same distinct first number of cells.

In some embodiments, the method for automated cell classificationfurther includes applying a plurality of second classifiers to eachimage to determine, for the each image, a second classificationprobability associated with each second classifier. Each secondclassifier is associated with a distinct third number of cells. The eachsecond classifier determines the second classification probability forthe each image based on the plurality of cell features, and furtherbased on one or more additional cell features associated with one ormore of the plurality of the first classification probabilitiesassociated with one or more images included in the plurality of imagesthat are temporally adjacent to the each image. The secondclassification probability indicates a second estimated likelihood thatthe distinct third number of cells associated with the each secondclassifier is shown in the each image, the each of the plurality ofimages thereby having a plurality of the second classificationprobabilities associated therewith. The method for automated cellclassification further includes classifying each image as showing afourth number of cells based on the distinct third number of cellsassociated with the each second classifier and the plurality of secondclassification probabilities associated therewith. In some embodiments,the plurality of images are a time-lapse series of images.

In some embodiments, the distinct third number of cells associated withthe each second classifier is selected from the group consisting of onecell, two cells, three cells, and four or more cells.

In some embodiments, each of the plurality of second classifiers isconfigured based on a second plurality of training images showing thedistinct third number of cells associated with the each secondclassifier.

In some embodiments, each of the second plurality of training images isdistinct from all of the first plurality of training images

In some embodiments, the distinct third number of cells associated withthe each second classifier is the same as the distinct first number ofcells associated with a corresponding one of the plurality of firstclassifiers

In some embodiments, the method for automated cell classificationfurther includes determining cell activity parameters of the one or morecells based on the fourth number of cells in the each image. In someembodiments, the determined cell activity parameters include one or moreof the following: a duration of first cytokinesis, a time intervalbetween cytokinesis 1 and cytokinesis 2, a time interval betweencytokinesis 2 and cytokinesis 3, a time interval between a first andsecond mitosis, a time interval between a second and third mitosis, atime interval from fertilization to an embryo having five cells, and atime interval between syngamy and the first cytokinesis.

In some embodiments, the method for automated cell classificationfurther includes applying a refining algorithm to the plurality ofimages to determine, based on the plurality of images, that one or moreof the plurality of images classified as showing the fourth number ofcells instead shows a fifth number of cells different from the fourthnumber of cells.

In some embodiments, the refining algorithm is a Viterbi algorithm.

In some embodiments, the method for automated cell classificationfurther includes determining cell activity parameters of the one or morecells based on the second number of cells in the each image. In someembodiments, determining cell activity parameters of the one or morecells based on the second number of cells in the each image

In some embodiments, the method for automated cell classificationfurther includes applying a predictive criterion to the one or morecells based on the determined cell activity parameters to determine apredicted outcome included in a plurality of specified outcomes. In someembodiments, the one or more cells shown in the plurality of images areselected from the group consisting of: a human embryo, one or moreoocytes, and one or more pluripotent cells.

In some embodiments, the plurality of first classifiers are AdaBoostclassifiers configured to perform binary classification.

In some embodiments, each of the plurality of cell features is one ormore of the following types: shape type, texture type, and edge type.

In some embodiments, at least one of the one or more machine learnedcell features is learned via unsupervised learning from a plurality oflearning images.

FIG. 11 illustrates a method 1100 for image-based embryo outcomedetermination, according to an embodiment of the invention.

At step 1110, a classifier is applied to a first time-lapse series ofimages of one or more cells to determine, for the first time-lapseseries of images, a classification probability. The classificationprobability can indicate an estimated likelihood that a first outcomefor development of the one or more cells is shown by the firsttime-lapse series of images. The first outcome is included in aplurality of outcomes for cell development associated with theclassifier.

At step 1120, the first time-lapse series of images can be classified asshowing the first outcome based on the plurality of outcomes associatedwith the classifier and the classification probability.

In some embodiments, the method can further comprise extracting afeature vector from the first time-lapse series of images, where thefeature vector is based on each of the first time-lapse series ofimages. The feature vector can include an element based on a frequencyof occurrence in each of the first time-lapse series of images of acodeword associated with a machine learned cell feature.

In some embodiments, the feature information is based on a featurevector extracted from one or more of the plurality of time-lapse seriesof images. The feature vector extracted from one or more of theplurality of time-lapse series of images can be based on each imageincluded in the one or more of the plurality of time-lapse series ofimages.

In some embodiments, the codeword associated with the machine learnedcell feature is extracted from the feature vector extracted from one ormore of the plurality of time-lapse series of images, and wherein thefeature information includes the codeword. In some embodiments, themachine learned cell feature is one or more of the following: edge type,texture type, and shape type. In some embodiments, the plurality ofoutcomes include one or more of the following pairs of outcomes:blastocyst and arrested; implantation and no implantation; and pregnancyand no pregnancy.

In some embodiments, the classification probability is a firstclassification probability, and the classifier is further configured todetermine additional classification probabilities based on featureinformation associated with each of the plurality of time-lapse seriesof images. In some embodiments, classifying the first time-lapse seriesof images is further based on the additional classificationprobabilities. In some embodiments, the first classification probabilityis greater than each of the additional classification probabilities.

In some embodiments, the classifier is an AdaBoost classifier. In someembodiments, the one or more cells in the first time-lapse series ofimages is of the same cell type as one or more cells in each of aplurality of time-lapse series of images, said cell type selected from:a human embryo, one or more oocytes, and one or more pluripotent cells.

In some embodiments, a method for image-based outcome determinationcomprises: applying a classifier to a first time-sequential series ofimages of one or more cells to determine, for the first time-sequentialseries of images, a classification probability. The classificationprobability indicates an estimated likelihood that a first outcome fordevelopment of the one or more cells is shown by the firsttime-sequential series of images. The first outcome is included in aplurality of outcomes for cell development associated with theclassifier.

In some embodiments, the method for image-based outcome determinationfurther includes classifying the first time-lapse series of images asshowing the first outcome based on the plurality of outcomes associatedwith the classifier and the classification probability.

In some embodiments, the method for image-based outcome determinationfurther includes extracting series feature information from the firsttime-sequential series of images, wherein the applying the classifier tothe first time-sequential series of images is based on the seriesfeature information.

In some embodiments, the series feature information is representative ofthe first outcome and is associated with an entirety of the firsttime-sequential series of images.

In some embodiments, the extracting the series feature informationincludes extracting local feature information associated with a portionof one or more of the first time-sequential series of images, anddetermining the series feature information based on the local featureinformation and a plurality of codewords.

In some embodiments, the determining the series feature informationincludes associating the local feature information with one or moreclusters, each of the one or more clusters being associated with acorresponding one of the plurality of codewords, and determining afrequency of occurrence of the one or more codewords across the firsttime-sequential series of images. The series feature informationincludes the frequency of occurrence of each of the one or morecodewords across the first time-sequential series of images.

In some embodiments, each of the plurality of codewords is associatedwith a cell feature that is one or more of the following: edge type,texture type, and shape type.

In some embodiments, each of the plurality of codewords is determinedfrom a plurality of unlabeled images of at least one cell throughunsupervised learning.

In some embodiments, the method for image-based outcome determinationfurther includes training the classifier based on series featureinformation associated with each of a plurality of time-sequentialseries of images, the each of the plurality of time-sequential series ofimages being associated with one of the plurality of outcomes. In someembodiments, the training the classifier includes extracting the seriesfeature information from the each of the plurality of time-sequentialseries of images. In some embodiments, series feature informationassociated with one of the plurality of time-sequential series of imagesis representative of an associated one of the plurality of outcomes, andis associated with an entirety of the one of the plurality of timesequential series of images. In some embodiments, the extracting theseries feature information includes extracting local feature informationassociated with a portion of one or more of the plurality oftime-sequential series of images, and determining the series featureinformation based on the local feature information and a plurality ofcodewords determined from a plurality of unlabeled images of at leastone cell through unsupervised learning.

In some embodiments, the determining the series feature informationincludes associating the local feature information with one or moreclusters, each of the one or more clusters being associated with acorresponding one of the plurality of codewords, and determining afrequency of occurrence of the one or more codewords across each of theone or more of the plurality of time-sequential series of images,wherein the series feature information includes the frequency ofoccurrence of each of the one or more codewords across the each of theone or more of the plurality of time-sequential series of images. Insome embodiments, each of the plurality of codewords is associated witha cell feature that is one or more of the following: edge type, texturetype, and shape type. In some embodiments, each of the plurality ofcodewords is determined from a plurality of unlabeled images of at leastone cell through unsupervised learning.

In some embodiments, the first time-sequential series of images is atime-lapse series of images. In some embodiments, the plurality ofoutcomes include one or more of the following pairs of outcomes: blastand arrested; implantation and no implantation; and pregnancy and nopregnancy. In some embodiments, the classifier is an AdaBoostclassifier. In some embodiments, the one or more cells is selected fromthe group consisting of: a human embryo, one or more oocytes, and one ormore pluripotent cells.

EXAMPLES Example 1

This example presents a multi-level embryo stage classification methodto estimate the number of cells at multiple time points in a time-lapsemicroscopy video of early human embryo development. A 2-levelclassification model is proposed to classify embryo stage within aspatial-temporal context. A rich set of discriminative embryo featuresare employed, hand-crafted, or automatically learned from embryo images.The Viterbi algorithm further refines the embryo stages with the cellcount probabilities and a temporal image similarity measure. Theproposed method was quantitatively evaluated using a total of 389 humanembryo videos, resulting in a 87.92% overall embryo stage classificationaccuracy.

INTRODUCTION

Timing/morpho-kinetic parameters measured from time-lapse microscopyvideo of human embryos, such as the durations of 2-cell stage and 3-cellstage, have been confirmed to be correlated with the quality of humanembryos and therefore can be used to select embryos with highdevelopmental competence for transfer to IVF patients. Accurately andobjectively measuring these timing parameters requires an automatedalgorithm that can identify the stage of human embryo (i.e. number ofcells) during a time-lapse imaging process. This example is focused onclassifying human embryos into four stages, i.e. 1-cell, 2-cell, 3-cell,and 4-or-more-cell. This problem can be challenging due to variations inmorphology of the embryos, occlusion, and imaging limitations.

This example presents a 3-level method to classify embryo stage intime-lapse microscopy video of early human embryo development. This workapplies machine learning techniques to classify human embryo stages forextraction of predictive parameters of clinical outcome. Theclassification method and learned embryo features (i.e. bag-of-features(BoF)) can be easily adapted to various imaging modalities, includingfor other cell classification and mitosis detection problems.

Methodology

FIG. 12 illustrates an exemplary approach for image-based cellclassification, in accordance with an embodiment of the invention. Givena human embryo video 1210 acquired with time-lapse microscopy, a richset of 62 standard hand-crafted features and 200 automatically learnedbag-of-features are extracted from each frame of the video. The level-1Adaboost classification model 1220 consists of 4 Adaboost classifierstrained for classifying one class from the rest classes using the 262features. Level-1 classification is performed using this classificationmodel on each frame independently. The level-2 Adaboost classificationmodel 1230 also consists of 4 Adaboost classifiers trained withaugmented feature set that includes both the 262 features and additionalfeatures computed from level-1 class probabilities. Level-2 Adaboost isdesigned to exploit local temporal context and refine the level-1classification results. At level 3 (see reference character 1240), theViterbi algorithm integrates prior knowledge, enforces thenon-decreasing number of cells, and generates the final embryo stageclassification results within the global context.

Embryo Features

The embryo features include 62 hand-crafted features (22 Gray-LevelCo-occurrence Matrices (GLCM), 10 Gabor features, and 5 regionproperties) and 200 Bag-of-Features learned automatically from embryoimage. The GLCM, LBP, and Gabor features are well-known texture featuresfor classification problems. Hessian features are statistics computedfrom the first eigenvalues of the Hessian-filtered images that enhancethe cell edges. The region properties (area, member of convex hallpoints, solidity, eccentricity, and perimeter) are computed from anrough embryo mask obtained by applying a shortest path algorithm toextract the embryo boundary in polar image space.

FIGS. 13A and 13B illustrate a bag of features in accordance with anexample, showing (a) examples of dense and sparse occurrence histogramsgenerated from sparsely detected descriptors and densely sampleddescriptors with a learned codebook; and (b) four examples of clusters(appearance codewords) generated by k-means clustering. The bag offeatures (BoF) is based on keypoint descriptors such as SIFT. Thisexample employs the basic SIFT descriptor to demonstrate theeffectiveness of BoF. Both densely sampled descriptors 1310A andsparsely detected descriptors 1310B are used in the method. K-meansclustering was employed to build a codebook with 200 codewords from SIFTdescriptors (128-dimension vectors) extracted from training embryoimages. Each of the clusters 1320A-1320D represents an intrinsic texturepattern of embryos, and its centroid is kept as one of the codewords.Given a testing image, descriptors are extracted first and thenquantized by hard-assigning each descriptor to one codeword. The finalBoF (1330A, 1130B) is an occurrence histogram that represents thefrequency of the codewords.

The additional level-2 features are temporal contextual featurescomputed from class-conditional probabilities output by level-1Adaboost. At each frame, the mean, median, max, min, and standarddeviation of the class-conditional probabilities of its localneighborhood (e.g. 5 frames) are computed and added to the originalfeature set.

2-Level Adaboost Classification Model

The one-vs-all scheme is employed to handle this multi-classclassification problem with binary Adaboost classifiers. Alternatively,the AdaBoost.M1 or Adaboost.M2 can also be used, which are multi-classextensions to Discrete Adaboost. There are four Adaboost classifiers ateach level of the 2-Level Adaboost classification model. Each Adaboostclassifier is consisted of a set of base stump classifiers and trainedto separate one class from the other classes. For a Adaboost classifiertrained for class iε{1,2,3,4}, its output of a image frame is

$\begin{matrix}{{P\left( {y_{t} = \left. i \middle| x_{t} \right.} \right)}\frac{\sum\limits_{k = 1}^{N}\;{a_{ik}{h_{ik}\left( x_{t} \right)}}}{\sum\limits_{k = 1}^{N}\; a_{ik}}} & (1)\end{matrix}$

where x_(t) is the extracted feature vector for frame t, a_(ik) is theweight of the base classifiers, h_(ik)ε{0,1} is the output of the baseclassifiers, and P(y_(t)=i|x_(t)) is the class-conditional probabilitynormalized to [0,1] (FIG. 12).

Temporal Image Similarity

Besides representing the embryo image in proposed method, the BoF isalso used to compute a temporal image similarity measure 1250 (FIG. 12)that is subsequently used by the Viterbi algorithm to define the statetransitional probability. Given the normalized BoF histograms of twoconsecutive embryo frames, the temporal image similarity at frame t isdefined based on the Bhattacharyya distance of these two histograms. Oneexample of the temporal image similarity is shown in FIG. 14. Thetemporal similarity measure based on BoF is registration free. Those“dips” in the plot are good indications of stage transition.

Global Embryo Stage Refinement

At level-3 of the proposed method, the Viterbi algorithm is employed torefine embryo stages within global context. The problem is to infer thebest state sequence of embryos that maximizes the posterior probabilityP(Y|X):

$\begin{matrix}{{\hat{Y} = {\arg{\max\limits_{y}{P\left( Y \middle| X \right)}}}},} & (2)\end{matrix}$

where, Y={y₁ . . . , y_(T)} is the state sequence, X={x₁, . . . , x_(T)}are the feature vectors representing the embryo images.

The Viterbi algorithm recursively finds the weight V_(t,i) of the mostlikely state sequence ending with each stage i at time t.

$\begin{matrix}{{V_{1,i} = {{P\left( {\left. x_{1} \middle| y_{1} \right. = i} \right)}{P\left( {y_{1} = i} \right)}}},} & \left( {3a} \right) \\{{V_{t,i} = {{P\left( {\left. x_{t} \middle| y_{t} \right. = i} \right)}{\max\limits_{j}\left( {{P\left( {y_{t} = {\left. i \middle| y_{t - 1} \right. = j}} \right)}V_{{t - 1},j}} \right)}}},{t \neq 1.}} & \left( {3b} \right)\end{matrix}$

where, P(y₁=i) represents the prior probability of each class at thefirst frame, P(x_(t)|y_(t)=i) is the observation probability, andP(y_(t)=i|y_(t-1)=j) is the transitional probability. Since an embryoalways starts with 1-cell stage, P(y_(t)=i) is set to 1 for i=1 and 0for the other stages. If it is assumed that the 4 stages are equallyprobable for the rest frames, the observation probabilityP(x_(t)|y_(t)=i) is simply the class-conditional probability output bythe level-2 Adaboost. The transitional probability P(y_(t)=i|y_(t-1)=j)is defined as a frame-dependent state.

Automated Embryo Stage Classification

$\begin{matrix}{{{Transition}\mspace{14mu}{matrix}\text{:}\mspace{14mu}{A(t)}} = \begin{pmatrix}{d(t)} & {1 - {d(t)}} & 0 & 0 \\0 & {d(t)} & {1 - {d(t)}} & 0 \\0 & 0 & {d(t)} & {1 - {d(t)}} \\0 & 0 & 0 & 1\end{pmatrix}} & (4)\end{matrix}$

where d(t) is the temporal image similarity defined in previous section.This transition matrix enforces non-decreasing number of cells andintegrates the temporal image similarity measure. When two consecutiveframes are almost the same (i.e. d(t) is close to 1), the transitionmatrix favors no embryo stage change.

Experimental Studies

To evaluate the performance of proposed classification method, humanembryo videos were collected from a variety of clinical sites and theclassification was evaluated based on classification accuracy and celldivision detection rate.

Dataset and Ground Truth

The video acquisition system consists of one inverted digitalmicroscope, which were modified for darkfield illumination. Embryoimages were acquired every 5 minutes for up to 2 days until the majorityof embryos reached the four-cell stage. The first 500 frames of eachembryo video were kept for analysis and each frame was cropped to a sizeof 151×151 pixels. The training data contains 327 human embryo videos(with 41741, 38118, 7343, and 69987 samples for each class respectively)and our testing data contains 389 human embryo videos (with 47063,48918, 9386, and 89133 samples for each class respectively) acquired atseveral clinical sites. Since the 3-cell stage is usually very short,fewer 3-cell training samples are used than the other classes.

Two human experts annotated frames when first cell division, second celldivision, and the third cell division occur. The ground-truth divisionframes are the average of annotations by the two human experts.Ground-truth for the embryo stage of each frame is converted from thecell division ground-truth.

Evaluation Results

The training dataset is split into two halves for training the level-1and level-2 Adaboost classifiers, respectively. Stump is used as baseclassifier and each Adaboost classifier contains 100 stumps.

In the first evaluation, the embryo stages predicted by proposed methodare compared with ground-truth. Overall classification accuracy andclassification accuracy for each class are shown for each level of themethod at Table 3. The confusion matrix for the final classificationresults is shown in Table 4. It can be seen from the results that eachlevel improves overall classification accuracy over the previous level.Over 90% 1-cell and 4-or-more-cell embryos have been classifiedcorrectly in the final results. Due to the lack of 3-cell trainingsamples and their resemblance to 2-cell and 4-or-more-cell embryos, only7.79% accuracy was reached by the level-1 Adaboost. The accuracy wasincreased to 10.71% by level-2 Adaboost and further improved to 20.86%by the level-3 Viterbi algorithm.

TABLE 3 Classification performance at different levels 1-cell 2-cell3-cell 4-or-more Overall Level-1 87.96% 77.45%  7.79% 85.03% 80.10%Level-2 88.04% 72.05% 10.71% 92.94% 82.53% Level-3 91.95% 85.58% 20.86%94.14% 87.92%

TABLE 4 Confusion matrix of the final classification result 1-cell2-cell 3-cell 4-or-more 1-cell 43276 (91.95%)  3399 (7.22%)  245 (0.52%) 143 (0.3%) 2-cell  643 (1.31%) 41866 (85.58%) 2518 (5.15%)  3891(7.95%) 3-cell   5 (0.05%)  4070 (43.36%) 1958 (20.86%)  3353 (35.72%)4-or-more   0 (0%)  2620 (2.94%) 2603 (2.92%) 83910 (94.14%)

In the second evaluation, the three division frames detected byclassification were compared with the three ground-truth embryo divisionframes. An estimated division frame is considered as a true-positive ifit is within certain tolerance to the ground-truth, and considered as afalse-positive otherwise. A ground-truth division frame is considered asfalse-negative if there is no predicted division frame within certaintolerance.

FIG. 15 illustrates exemplary results for (a) precision rate and (b)recall rate of cell division detection as a function of offset toleranceobtained from an exemplary 3-level classification method, in accordancewith an embodiment of the invention. The precision and recall curves forthree subsets of features were generated to evaluate their contributionsto the classification performance separately. It can be seen from FIG.15 that BoF outperformed the handcrafted features (RegionProp andGLCM+LBP+Hessian+Gabor, described with reference to Table 1), and thatthe combination of BoF and the handcrafted features reached the highestperformance. For example, at 10-frame tolerance, a precision of 84.58%and a recall rate of 75.63% were achieved by the combined feature set.

This Example presents a classification method for effectivelyclassifying embryo stages in time-lapse microscopy of early human embryodevelopment. When applied to a large testing dataset collected frommultiple clinical sites, the proposed method achieved a total of 87.92%classification accuracy.

Example 2

As noted earlier, some aspects of the invention are also operable fortracking of cell activity. Accordingly, some aspects of the inventionare operable for automated, non-invasive cell activity tracking. In someembodiments, automated, non-invasive cell activity tracking is fordetermining a characteristic of one or more cells without invasivemethods, such as injection of dyes. The cell activity tracking can beapplied to one or more images of one or more cells. The images can be atime-sequential series of images, such as a time-lapse series of images.The cell(s) shown in the plurality of images can be any cell(s) ofinterest. For example, the cells can be a human embryo that may have oneor more cells. Other examples of such cells of interest include, but arenot limited to, oocytes and pluripotent cells.

In some embodiments, a number of the cells in each image is of interest,and can be determined by an embodiment of the invention. For example,the number of cells can be representative of an embryo at one or more ofthe one cell stage, the two cell stage, the three cell stage, the fourcell stage, and so on. In some embodiments, the four cell stagerepresents four or more cells. Alternatively or in addition, a geometryof the cells in each image is of interest, and can be determined by anembodiment of the invention. The geometry of the cells may include ashape of the cells and/or an arrangement of the cells.

In some embodiments, one or more of these characteristics of the cellsmay be determined by selecting one of multiple hypotheses per image. Theselected hypotheses may be the most likely sequence of hypotheses acrossa time-sequential series of images, and may include a set of shapes thatbest fit observable geometric characteristics (geometric features shownin one or more of the images) of the cells. In one embodiment, thegeometric features may include boundary information associated with eachof the one or more cells, such as boundary points and/or boundarysegments. Each boundary point and/or boundary segment may be mapped to aspecific cell (or to no cells). This mapping may be explicit orimplicit. Alternatively or in addition, shapes may be fit to theboundary points and/or boundary segments associated with each cell.These shapes may be ellipses, or other suitable shapes such as b-splinesor other smooth shapes. It will be understood that in thisspecification, references to shapes being fit to boundary segments canalso refer to shapes being fit to boundary points and/or other geometricfeatures associated with each of the one or more cells. In one example,the hypotheses may be selected based on multiple hypothesis inference,such as a data driven approximate inference.

The multiple hypotheses per image each include an inferredcharacteristic of the cells, such as an inferred number of the cellsand/or an inferred geometry of the cells. The multiple hypotheses perimage can be based on geometric features of the cells shown in theimage. There may be a mapping of a representation of each cell to one ormore boundary points and/or boundary segments associated with each cell.This mapping may be explicit or implicit. Alternatively or in addition,shapes may be fit to the boundary points and/or boundary segmentsassociated with each cell without generation of an explicit mappingbetween cells and boundary points and/or boundary segments associatedwith each cell. In this specification, references to boundary segmentsof cells and operations involving those segments (such as mapping,generation, merging, etc.) are examples of particular embodiments, anddo not limit the scope of the invention to embodiments that involveboundary segments.

In one embodiment, the cell boundary segments are an example ofobservable geometric information that can be determined based on theimages of the cells. Another example is cell boundary points. In oneembodiment, each of the multiple hypotheses per image can be viewed asrepresenting the inferred number and/or the inferred geometry of thecells through cell boundary feature labels that represent the mapping(inferred by each hypothesis) of the representation of each cell to theone or more boundary segments associated with each cell. The cellboundary feature labels may be cell boundary segment labels.Advantageously, the solution space across the multiple hypotheses perimage is over the discrete set of cell boundary segment labels, which isa much smaller solution space than the continuous set of parametersrepresenting all possible groups of shapes of a particular type thatcould represent the cells. For example, for tracking up to 4 cells withellipses each defined by 5 continuous parameters (for example, majoraxis length, minor axis length, x-coordinate of the ellipse center,y-coordinate of the ellipse center, and yaw angle), the solution spacehas 20 continuous dimensions. In contrast, the label for each of Kboundary segments may have one of five discrete values (for example, 0for assignment to none of the cells, or 1-4 for assignment to a specificone of the four cells), for a total of only 5^(K) possible solutions.This significantly reduces the solution space by leveraging observablecell boundary segment information from the images, making hypothesisselection more tractable and reliable.

In some embodiments, once the cell boundary segments are labeled, thecell boundary segments can be grouped together and fit to shapes otherthan ellipses, such as more complex shapes represented by a largernumber of continuous parameters. For example, blastomeres within embryoscan deviate significantly from ellipses. Advantageously, thedimensionality of the solution space over the discrete set of cellboundary labels is unchanged (in the example above, still 5^(K) possiblesolutions, where there are K boundary segments). This is unlike thedimensionality of the solution space over the continuous set ofparameters representing all possible groups of shapes of a particulartype that could represent the cells, which increases if the number ofcontinuous parameters defining the shape increases.

In some embodiments, by solving for cell boundary segment labels, cellboundary segments can be assigned to none of the cells. Advantageously,this can allow for a more robust treatment of outliers and falsepositive boundaries, which is a common problem associated withprocessing of cell boundary data.

In some embodiments, based on the characteristics of the cellsdetermined based on hypothesis selection per image, parameters relatedto embryo health and or fate (outcome, such as whether an embryo isexpected to reach blastocyst or arrest) can be determined. Theseparameters may include but are not limited to one or more of a durationof first cytokinesis, a time interval between cytokinesis 1 andcytokinesis 2, a time interval between cytokinesis 2 and cytokinesis 3,a time interval between a first and second mitosis, a time intervalbetween a second and third mitosis, a time interval from fertilizationto an embryo having five cells, and a time interval from syngamy to thefirst cytokinesis. From one or more of these parameters, an indicator ofdevelopment competence of the embryo for implantation into a femalehuman subject can be determined in an automated, non-invasive fashion.

Aspects of the invention are also operable for automated, non-invasivecell activity tracking in conjunction with tracking-free approaches suchas classification and/or interframe similarity determination to enhancedetermination of cell/embryo characteristics related to embryo healthand/or fate/outcome.

FIG. 16 illustrates a non-limiting example of an automated cell trackingapproach applied to images of cell development such as embryodevelopment, in accordance with an embodiment of the invention. A seriesof time-sequential images 1602 ₁ to 1602 ₄₀₀ (1602 ₁, 1602 ₁₅₀, 1602₂₅₀, and 1602 ₄₀₀ shown) shows development of one or more cells 1600_(1/1), 1600 _(150/1) to 1600 _(150/c), . . . 1600 _(400/1) to 1600_(400/c) shown in each of the images 1602 ₁, 1602 ₁₅₀, 1602 ₂₅₀, and1602 ₄₀₀, respectively (where c is the number of cells shown in theimage 1602 _(i), and c can be a different value for each image 1602_(i)). In this example, the one or more cells 1600 are included in ahuman embryo. The subscript 1 to 400 is an identifier associated witheach individual image 1602 _(i), where the identifier may be a timeindicator, a frame number, or another suitable identifier fordistinguishing the images 1602 and their contents. In this example, forimage 1602 ₁, one cell 1600 _(1/1) is shown. For image 1602 ₁₅₀, twocells 1600 _(150/1) to 1600 _(150/2) are shown. For image 1602 ₂₅₀,three cells 1600 _(250/1) to 1600 _(250/3) are shown. For image 1602₄₀₀, four cells 1600 _(400/1) to 1600 _(400/4) are shown.

After determination of cell boundary segments, the identified cellboundary segments 1604 _(1/1), 1604 _(150/1) to 1604 _(150/k), . . .1604 _(400/1) to 1604 _(400/k) are shown for each of the images 1602 ₁,1602 ₁₅₀, 1602 ₂₅₀, and 1602 ₄₀₀, respectively (where k is the number ofcell boundary segments determined to be in the image 1602 _(i), and kcan be a different value for each image 1602 _(i)). For clarity, thecell boundary segments 1604 are not overlaid on the images 1602, andadjacent cell boundary segments 1604 are shown with different line typesand thicknesses.

One or more hypotheses 1606 _(1/1), 1606 _(150/1) to 1606 _(150/n), . .. 1606 _(400/1) to 1606 _(400/n) are shown per image for each of theimages 1602 ₁, 1602 ₁₅₀, 1602 ₂₅₀, and 1602 ₄₀₀, respectively (where nis the number of hypotheses for the image 1602 _(i), and n can be adifferent value for each image 1602 _(i)). In this example, each of then hypotheses 1606 _(i/1) to 1606 _(i/n) per image 1602 _(i) is based ona mapping of a representation of each cell 1600 _(i/c) to one or more ofthe boundary segments 1604 _(i/1) to 1604 _(i/k) associated with eachcell 1600 _(i/c) for that hypothesis. In other embodiments, each of thehypotheses 1606 may be based on other types of observable geometricinformation such as boundary points. In other embodiments, an explicitmapping of cells to boundary points and/or boundary segments is notrequired. In this example, each hypothesis 1606 includes inferredcharacteristics of the one or more cells 1600 including an inferrednumber of one or more cells 1602 and an inferred geometry of the one ormore cells 1600. For example, the inferred number of cells 1600associated with the hypothesis 1606 _(400/1) is four (the number ofellipses 1610 associated with the hypothesis 1606 _(400/1)), and theinferred geometry of the one or more cells 1600 associated with thehypothesis 1606 _(400/1) is indicated by one or more of the shape andarrangement of the ellipses 1610. In another example, the ellipses 1610may be another suitable shape, such as a b-spline or another smoothshape. Note that since there is only one cell 1600 _(1/1) and one cellboundary segment 1604 _(1/1) determined to be shown by the image 1602 ₁,there may be only one hypothesis associated with the image 1602 ₁: thehypothesis 1606 _(1/1), mapping the cell 1600 _(1/1) to the cellboundary segment 1604 _(1/1). Alternatively or in addition, there may bea second hypothesis associated with the image 1602 ₁: a hypothesis (notshown) mapping none of the cells to the cell boundary segment 1604_(1/1).

In this example, the hypotheses 1612 _(i/n) (including hypotheses 1606_(1/1), 1606 _(150/2), 1606 _(250/2), and 1606 _(400/1) in this example)are selected. Characteristics 1608 of the cells 1600 associated with theselected hypotheses 1606 _(1/1), 1606 _(150/2), 1606 _(250/2), and 1606_(400/1), including the number of cells 1600 and the geometry of the oneor more cells 1600 associated with each selected hypothesis 1606 _(1/1),1606 _(150/2), 1606 _(250/2), and 1606 _(400/1), are shown for each ofthe images 1602 ₁, 1602 ₁₅₀, 1602 ₂₅₀, and 1602 ₄₀₀, respectively. Forclarity, the cell characteristics 1608 are not overlaid on the images1602.

FIG. 16B illustrates an expanded view of the cell boundary segments 1604shown in FIG. 16A, in accordance with an embodiment of the invention.The identified cell boundary segments 1604 _(1/1), 1604 _(150/1) to 1604_(150/k), 1604 _(400/1) to 1604 _(400/k) are shown for each of theimages 1602 ₁, 1602 ₁₅₀, 1602 ₂₅₀, and 1602 ₄₀₀, respectively. Forclarity, adjacent cell boundary segments 1604 are cross-hatched withdifferent patterns. In this example, portions 1610 of the cellboundaries shown with solid black fill are occluded portions of the cellboundaries that are not included in the cell boundary segments 1604.

FIG. 17A illustrates a non-limiting example of a cell tracking framework1700, in accordance with an embodiment of the invention. The celltracking framework 1700 may be associated with human embryo development.The tracking framework 1700 may be based on a probabilistic graphicalmodel (PGM) 1702 which captures relevant unknowns, such as cell boundaryfeatures, cell boundary feature labels that represent a mapping of therepresentation of each cell to the one or more boundary segmentsassociated with each cell, cell geometry (such as cell shape), celldivision events, and/or number of cells over time. In some embodiments,and referring to FIG. 16A, the PGM 1702 is a chain graph that may span atime interval over which the images 1602 are taken, and each node in thegraph is a variable. The PGM 1702 may be a conditional random field(CRF) that represents a stochastic evolution of elliptical cells. A linkbetween two variables in the PGM 1702 signifies a direct statisticaldependence between them. The PGM 1702 includes nodes 1702 _(1/o) to 1702_(400/o) that represent information (evidence) observable from theimages 1602 ₁ to 1602 ₄₀₀, respectively, and nodes 1702 _(1/i) to 1702_(400/i) that represent variables associated with the images 1602 ₁ to1602 ₄₀₀, respectively, to be inferred based on cell trackingRepresentative nodes 1702 _(3/i) and 1702 _(3/o) of the PGM 1702 areexpanded to illustrate, for one time slice, exemplary underlying nodesthat may be included in the PGM 1702.

In one embodiment, observable evidence associated with the node 1702_(3/i) may include cell boundary segment features observed from theimage 1602 ₃, and represented by one or more segment nodes 1704. Thesegment nodes 1704 represent segments s^((t))={S_(k) ^((t)) k=1 . . .K_(t) where S_(k) ^((t)) is a collection of points S_(k,i) ^((t))ε

² with iε{1 . . . m_(k) ^((t))}. At each frame t there are K_(t)segments, each with m_(k) ^((t)) points, kε{1 . . . K_(t)}. Variables tobe inferred that are associated with the node 1702 _(3/o) may includesegment to cell labels represented by one or more label nodes 1706,shape descriptors (in this example, ellipse descriptors) represented byone or more ellipse descriptor nodes 1708, number of cells representedby number of cells node 1710, and cell division events represented byone or more cell division event nodes 1712. The label nodes 1706represent labels assigning segments to cells l^((t))ε{0, 1 . . .N_(max)}^(K) ^(t) , where in this example N_(max)=4 cells. The ellipsedescriptor nodes 1708 represent ellipses e_(n) ^((t))ε

⁵, nε{1 . . . N_(max)}. The number of cells node 1710 represent numberof cells N^((t))ε{1 . . . N_(max)}. The cell division event nodes 1712represent division event d^((t))ε{0,1}. Each ellipse e_(n) ^((t)) may beassociated with its parent, e_(Pa(n)) ^((t-1)).

The PGM 1702 captures at least three types of context: (1) intracellshape and geometric coherence that relate the boundary pixels of anygiven cell to each other; (2) intercell geometric context that relatesthe shapes of different cells within an embryo to each other; and (3)temporal context relating shape and topology between image frames 1602(see FIG. 1A). Intracell shape and geometric coherence refers to therelationship between boundary points that belong to one cell, andrelates to, for example, segment generation and constrained segmentmerging (see FIG. 18A). Intercell geometric context refers to therelationship of shapes of different cells (such as in an embryo) to oneanother. For example, a very large cell and a very small cell are notlikely contained in the same embryo, and hypotheses containing the verylarge cell and the very small cell can be rejected (or scored low).Temporal context refers to, for example, cell shape deformation withtime and cell division.

Example 3 below illustrates in detail an example of the mathematicalform of the joint probability distribution over the variablesrepresented in the PGM 202, and that discussion is not repeated here.Eq. (8) (see Example 3) illustrates an example of the observation modelincluded in the joint probability distribution shown in Eq. (7) (seeExample 3). The exemplary observation model of Eq. (8) (see Example 3)is generalized to include information associated with tracking-freeapproaches such as classification and/or interframe similaritydetermination that may be used in conjunction with cell activitytracking. When cell activity tracking is used without tracking-freeapproaches, an example of the observation model is shown below as Eq.(5), with the term c_(N) ^((t))(N^((t))) associated with the classifierset to zero and the term δ^((t)) associated with interframe similarityset to 0.5 (so that a division event is equally likely or unlikelybetween adjacent images 1602 (see FIG. 16A)). φ₂(d^((t)),δ^((t))),thereby becomes a constant and can be dropped from the observationmodel.Φ(e ^((t)) ,l ^((t)) ,N ^((t)) ,d ^((t)) ,s ^((t))),=φ₀(e ^((t)))(φ₁(e^((t)) ,l ^((t)) ,N ^((t)) ,s ^((t)))  (5)

In Eq. (8) (see Example 3), the termφ₁(e^((t)),l^((t)),N^((t)),s^((t))), which encodes compatibility ofellipses, segments, and labels, captures intracell shape and geometriccoherence that relate segments s^((t)) to cells. The term φ₀(e^((t)),which encodes geometric constraints, captures intercell geometriccontext that relates the shapes of different cells within an embryo toeach other. The motion (transition) model ψ(e^((t-1:t)),N^((t-1:t)),d^((t)))_(t=2 . . . T), shown in Eqs. (7) and (11) (seeExample 3), captures temporal context relating shape and topologybetween image frames.

FIG. 17B illustrates a non-limiting example of a cell tracking framework1720, in accordance with another embodiment of the invention. In oneembodiment, the cell tracking framework 1720 may be associated withhuman embryo development. Various aspects of the cell tracking framework1720 are similar to aspects of the cell tracking framework 1700described with reference to FIG. 17A, and those aspects are not repeatedhere. A PGM 1722 is in many respects similar to the PGM 1702 describedwith reference to FIG. 17A, except that the PGM 1722 further includesadditional observable information. This additional observableinformation may include an image similarity measure δ^((t))ε[0,1]represented by one or more image similarity measure nodes 1724, and/or aclassifier on the number of cells c_(N) ^((t))ε

^(N) ^(max) represented by one or more classifier nodes 1726. The imagesimilarity measure may relate to a likelihood of occurrence of one ormore cell division events between adjacent images 1602 (see FIG. 16A).The classifier may be an AdaBoost or Support Vector Machine (SVM)classifier, may be single-level or multi-level, and may estimateposterior probabilities of number of cells (in one embodiment, c_(N)^((t)) in Eq. (8) (see Example 3) below) from a set of hand-craftedand/or machine learned discriminative image features. Such a classifiercan be configured to perform image-based cell classification asdisclosed earlier.

In one embodiment, cell activity tracking can be used in conjunctionwith tracking-free approaches such as classification and/or interframesimilarity determination, as described above with reference to FIG. 17B.Example 3 below illustrates in detail an example of the mathematicalform of the joint probability distribution over the variablesrepresented in PGM 222, and that discussion is not repeated here.

FIG. 18A illustrates a method 300 for obtaining cell boundary featureinformation, in accordance with an embodiment of the invention. Thiscell boundary feature information may include cell boundary segments1604 (see FIG. 1). In one embodiment, the cell boundary segments 1604may be boundary segments of one or more cells included in a humanembryo. For each image 1602 (see FIG. 16), boundary points of thecell(s) are determined (block 1802). Cell boundary segments are thengenerated based on the cell boundary points (block 1804). One or morepairs of the cell boundary segments may then be merged (block 1806) intothe cell boundary segments 1604. Segment merging aims to combine thegenerated cell boundary segments (from block 1804) into a smaller set oflonger segments in order to reduce the total number of combinations formapping of a representation of each of one or more cells to one or moreboundary segments associated with each of the one or more cells. Thesepotential mappings can have associated segment to cell labelsrepresented by one or more label nodes 1706 (see description withreference to FIG. 17A), and are observable evidence from the images 1602that can be leveraged as part of reducing the number of hypotheses to beconsidered during hypothesis selection (see description with referenceto FIG. 18B).

With reference to extraction of cell boundary points (block 1802), inone embodiment, boundary points can be extracted using a Hessianoperator, which provides a boundary strength and orientation angle foreach pixel of each image 1602 (see FIG. 16). The Hessian operator may berepresented as a matrix of second-order partial derivatives. Theboundary strength at each pixel of each image 1602 may be obtained basedon the eigenvalues of this matrix, and the orientation angle at eachpixel of each image 1602 may be obtained based on the eigenvectors ofthis matrix. In one embodiment, the Hessian images resulting fromapplication of the Hessian operator to each image 1602 may bethresholded. The effect of applying the Hessian operator to each image1602 followed by thresholding can be to emphasize contrast between cellboundary points and other pixels within the images 1602, whetherinternal to or external to the cells 1600 (see FIG. 16). In otherembodiments, other approaches for boundary point extraction can be used,including but not limited to intensity gradients (for example, Cannyedge detection and/or Sobel edge detection), texture gradients, regionbased approaches, and/or other suitable approaches known to one ofordinary skill in the field of computer vision.

With reference to generation of boundary segments (block 1804), in oneembodiment, the boundary segments can be generated through a directedlocal search for coherent boundary pixels included in the set ofextracted cell boundary points. As described previously, boundarysegment generation is based on intracell shape and geometric coherencethat relate the boundary pixels of any given cell to each other. Forexample, boundary points that essentially lie along an elliptical cellboundary and essentially cover the elliptical boundary can be consideredto be highly coherent and compatible with that cell. On the other hand,randomly scattered points are incoherent and not compatible with anyparticular cell. The cell shape in this case is assumed to be anellipse, but other suitable shape models (such as but not limited tob-splines) can also be assumed. In one embodiment, the generation ofboundary segments and the mapping of the boundary segments torepresentations of cells occurs in a bottom up fashion. Boundarysegments can be determined by searching for points that lie along ornear a smooth curve. If these points continue along a complete ellipse,the boundary segment is the same as the ellipse. But cell boundaries canalso be broken and discontinuous (such as due to occlusion by othercells), so after detecting segments the mapping of the boundary segmentsto representations of cells typically occurs.

In one embodiment, the boundary points can be grouped into boundarysegments subject to the following two competing criteria: (1) create asfew segments as possible; and (2) associate each segment with at mostone cell in the image 1602. In other words, in one embodiment, boundarysegment generation aims to group the initial boundary points into as fewsegments as possible, but errs on the side of breaking apart segmentswhen unsure as to whether they represent the same cell. The subsequentsegment merging (block 1806) aims to resolve these ambiguities.

In one embodiment, the boundary segments can be generated through ridgesearch segment generation. A ridge search seeks a path along whichconsecutive peaks occur. An analogy for the ridge search is walkingalong the top of a mountain chain and seeking the next peak along thedirection of that chain. This search can be performed on a Hessian imagegenerated through boundary point extraction (block 1802) from the image1602. The ridge search starts by finding the strongest valued pixel inthe Hessian image as an entry point into a ridge. It then continues byprogressing along a trajectory that starts from the original pixel alongthe Hessian orientation angle for each pixel generated through boundarypoint extraction (block 1802) from the image 1602. It searches foranother high valued pixel along this trajectory, and starts over. It canrepeat this process until either there are no high value pixels in theexpected regions, or if the found high value pixel has an orientationangle that is too different than the current orientation angle, whichcan indicate an endpoint for the segment. When a segment's ridge searchis finished, a new ridge search is begun. This process is continueduntil all high value Hessian image pixels have been covered.

In other embodiments, other approaches for boundary segment generationcan be used, including but not limited to a breadth first search on theboundary points, ordering the boundary points in a minimal spanning treeand then breaking the tree at points of discontinuity, and/or othersuitable approaches known to one of ordinary skill in the field ofcomputer vision.

With reference to merging of boundary segments (block 1806), segmentmerging aims to combine the generated boundary segments (block 1804)into a smaller set of longer segments in order to reduce the totalnumber of combinations for mapping of segments to cells. In oneembodiment, for any two segments, segment merging may be based on one ormore of four criteria: (1) relative fit error; (2) continuity ofendpoints; (3) continuity of angle; and (4) curvature consistency. Therelative fit error criterion can involve fitting three curves, one foreach of the two input segments, and one for the merged segment. If thefit error of the merged segment is better than that of the individualinput segments, the likelihood of merging increases. The continuity ofendpoints criterion looks at how closely the two segments to be mergedare to each other if they were to be continued. Closer distance makes amerge more likely. The continuity of angle criterion is based on asimilar concept, except that it is based on the angle at the join pointfor the merged segment as well as the angle for each of the individualsegments were they to continue to the join point. The closer theseangles are to each other, the more likely a merge is. The curvatureconsistency criterion can be that if the mean curvature of the twosegments to be merged are close to each other, the more likely a mergeis.

In one embodiment, the segments can be merged (block 1806) based on amerging inference that analyzes geometric properties of the generatedboundary segments (block 1804) to determine if they can be merged into asmaller set of larger segments. The merging of the boundary segments canbe formulated as a graph partitioning on a graph whose vertices aresegments and whose edges indicate merging of segments, where the numberof partitions is unknown in advance.

FIG. 18B illustrates a method 1810 for generating a mapping of arepresentation of cells 1600 (see FIG. 16) to cell boundary featureinformation and refining hypotheses each including an inferredcharacteristic of one or more of the cells 1600, in accordance with anembodiment of the invention. In one embodiment, the cells 1600 may beincluded in a human embryo. At each image 1602 (see FIG. 16), hypothesesassociated with embryo development are generated, each of which isassociated with cell boundary segment labels that map representations ofeach of the cells 1600 to one or more of the cell boundary segments1604. At each image 1602 _(i), a number of “parent” hypotheses 1811selected from hypotheses 1606 _(i−1,n) for cells 1600 _(i−1/n) from theprevious image 1602 _(i−1) can be used to determine preliminaryhypotheses 1813 associated with the image 1602 _(i) (block 1812). One ormore of the inferred characteristics included in the preliminaryhypotheses 1813 associated with the image 1602 _(i), such as inferredgeometric parameters associated with ellipses associated with each ofthese preliminary hypotheses 1813, may be generated by sampling andperturbing ellipses associated with one or more of the parent hypotheses1811. In one embodiment, there may be one parent hypothesis 1811associated with each number of cells (such as 1, 2, 3, and 4 cells) thatcan be shown in the image 1602 _(i−1). Alternatively, there may be moreor fewer parent hypotheses 1811 associated with each number of cellsthat can be shown in the image 1602 _(i−1). In one embodiment, there maybe one, two, three or four preliminary hypotheses 1813. Alternatively,there may be a larger number of preliminary hypotheses 1813. At aninitial image 1602 ₁, an initial hypothesis may be generated by findingan ellipse that best fits boundary segments for the cell 1600 _(1/1).

In one embodiment, one or more detected segments can be assigned to norepresentation of any of the cells 1600. Advantageously, this can allowfor a more robust treatment of outliers and false positive boundaries,which is a common problem associated with processing of cell boundarydata.

Next, hypotheses 1815 are generated from the preliminary hypothesesbased on observable geometric information from the current image (image1602 _(i)) (block 1814). In one embodiment, the hypotheses 1815 may begenerated (block 1814) through expectation maximization (EM)optimization to obtain a data driven refined hypothesis based on atleast observable geometric information from the image 1602 _(i). Theobservable geometric information from the image 1602 _(i) may includeone or more of the shape and arrangement of the cells 1600 _(i/n) shownin the image 1602 _(i). The shape of the cells 1600 _(i/n) may becharacterized by multiple shape parameters. For example, for a shapethat is an ellipse, the shape parameters may include, but are notlimited to, major axis length, minor axis length, x-coordinate of theellipse center, y-coordinate of the ellipse center, and yaw angle. Thearrangement of the cells 1600 _(i/n) may be characterized by parametersrelated to, but not limited to, one or more of orientation of, locationof, and overlap between one or more of the cells 1600 _(i/n).Advantageously, by taking into account the observable geometricinformation from the current image 1602 _(i) as well as past images 1602₁ to 1602 _(i−1), the hypotheses 1915 may be refined to more closelytrack the full set of available, observable geometric information,thereby making hypothesis selection more tractable and reliable.

In one embodiment, the generation of the hypotheses 1915 in block 1814may include one or more of blocks 1816, 1818, and 1820. At the image1602 _(i), a mapping of a representation of each of the cells 1600_(i/n) associated with each of the preliminary hypotheses 1813 toboundary segments 1604 _(i/k) obtained from segment merging (block 1806)applied to the image 1602 _(i) may then be generated (block 1816). Inone embodiment, this mapping may be obtained by assigning each of thesegments 1604 _(i/k) to the closest shape (such as an ellipse) includedin each of the preliminary hypotheses 1813. For example, the ellipse towhich the average distance across all points in a segment 1604 _(i/k) issmallest can be the corresponding ellipse for the segment 1604 _(i/k).These mappings may be represented by the segment to cell labelsrepresented by the one or more label nodes 1706 (see FIG. 17).

Next, each of the preliminary hypotheses 1813 may then be refined basedon the mapping from block 1816 to obtain refined hypotheses 1815 at theimage 1602 _(i) (block 1818). Ideally, the entire boundary of each ofthe cells 1600 _(i/n) shown in the image 1602 _(i) would be visible, sothe boundary segments 1604 _(i/k) mapped to the preliminary hypotheses1813 would cover the entire boundary of each of the cells 1600 _(i/n).However, in a more typical scenario, sections of the boundaries of oneor more of the cells 1600 _(i/n) shown in the image 1602 _(i) may not bevisible, and may therefore effectively be missing. An estimate (such asan expected value) may need to be generated for these sections. In oneembodiment, portions of each ellipse associated with each preliminaryhypothesis 1813 are identified that do not have any data points nearbythat are associated with boundary segments 1604 _(i/k) mapped to eachpreliminary hypothesis 1813. In one embodiment, a number of equallyspaced points (such as 50 to 100, or any other suitable number) can begenerated from a parametric representation of each ellipse associatedwith each preliminary hypothesis 1813. Each of these points that doesnot have a data point sufficiently nearby that is associated withboundary segments 1604 _(i/k) mapped to each preliminary hypothesis 1813can be included in the ellipse as an estimated data point.

The refinement of the preliminary hypotheses 1813 to obtain refinedhypotheses 1815 at the image 1602 _(i) (block 1818) may then includefitting of a shape (such as but not limited to an ellipse) to each groupof boundary segments 1604 _(i/k) with the same segment to cell label(represented by the one or more label nodes 1706 (see FIG. 17)). Eachrefined hypotheses 1815 includes one or more of these newly fittedellipses, each ellipse being associated with an associated cell 1600_(i/n) characterized by the refined hypothesis 1815.

Next, each refined hypothesis 1815 may be scored based on the observablegeometric information from the image 1602 _(i) (block 1820), includingbut not limited to the boundary segments 1604 _(i/k) determined from thecells 1600 _(i/n) shown in the image 1602 _(i). In one embodiment, toobtain each refined hypothesis 1815, blocks 1816, 1818, and 1820 may berepeated until the fit quality (fit error) converges or a maximum numberof iterations is reached. Multiple refined hypotheses 1815 can begenerated at each image 1602 _(i). For example, a representative valueof the number of refined hypotheses 1815 generated at a given image 1602_(i) is in the range from 50 to 200, though more or fewer may begenerated.

In one embodiment, particle scoring criteria for a given frame include,but are not limited to, the fit quality (fit error) and coverage. Thefit quality (which can range from 0 to 1) and/or fit error (which canrange from 0 to infinity) indicate how well the cell boundary pointsassociated with each cell 1600 _(i/n) characterized by the refinedhypothesis 1815, including any estimated data points generated formissing portions of cell boundaries, fit the fitted shape (such as butnot limited to an ellipse) to each cell 1600 _(i/n). The coverageindicates how well the boundary of the fitted shape is covered by thecell boundary points associated with each cell 1600 _(i/n) characterizedby the refined hypothesis 1815, including any estimated data pointsgenerated for missing portions of cell boundaries. In one example, oneor more parameters associated with the coverage can range from 0 to 1,where 0 can mean no coverage, and 1 can mean full coverage, or viceversa. In addition, other parameters associated with the coverage cancharacterize inlier coverage, which is the ratio of the cell boundarypoints associated with each cell 1600 _(i/n) characterized by therefined hypothesis 1815, including any estimated data points generatedfor missing portions of cell boundaries, that are considered inliers tothe fitted shape. For example, one or more of these cell boundary pointsmay be excluded if they are too far away from the fitted shape. Whenthat happens, the inlier coverage can be accordingly reduced.

Next, parent hypotheses 1817 are selected for the image 1602 _(i+1) fromthe refined hypotheses 1815 (block 1822). In one embodiment, there maybe one parent hypothesis 1817 associated with each number of cells (suchas 1, 2, 3, and 4 cells) that can be shown in the image 1602 _(i).Alternatively, there may be more or fewer parent hypotheses 1817associated with each number of cells that can be shown in the image 1602_(i). The collection of the refined hypotheses 1815 and their scores areused to approximate a distribution over the refined hypotheses 1815,which is then marginalized to obtain an approximate distribution overthe number of cells. This marginal distribution can then used to selectthe parent hypotheses 1817. For example, the parent hypotheses 1817 maybe selected based on one or more of the following determined based onimages 1602 ₁ to 1602 _(i): an approximate max marginal measure ofnumber of cells at each of the images 1602 ₁ to 1602 _(i), anapproximate joint distribution over number of cells at each of theimages 1602 ₁ to 1602 _(i), and/or a marginal distribution over numberof cells at each of the images 102 ₁ to 102 _(i). These distributionsare described further with reference to FIGS. 18C,19A, and 19B.

FIG. 18C illustrates a method 1830 for selecting hypotheses 1612 fromthe hypotheses 1606 (see FIG. 16A), in accordance with an embodiment ofthe invention. FIGS. 19A-19B illustrate exemplary approaches forselection of the hypotheses 1612 for the images 1602 of FIG. 16A, inaccordance with embodiments of the invention. In one embodiment, theselection of the hypotheses 1612 is an approximate inference over thePGM 1702 of FIG. 17A. Alternatively, the selection of the hypotheses1612 is an approximate inference over the PGM 1722 of FIG. 17B.Alternatively, the selection of the hypotheses 1612 may be anapproximate inference over any suitable probabilistic graphical model.Referring to FIGS. 18C, 19A, and 19B, in one embodiment, approximate maxmarginal measures 1902 of number of cells at each of the images 1602 ₁to 1602 _(N) can be determined (block 1832) based on the refinedhypotheses 1815 (see FIG. 18B) for the images 1602 ₁ to 1602 _(N). Inthis example, the approximate max marginal measures 1902 are for 1 cell(1902A), 2 cells (1902B), 3 cells (1902C), and 4 cells (1902D). Thevalue of the approximate max marginal measures (y-axis) is plottedagainst image frame number (1 to 400). Then, an approximate jointdistribution over number of cells at each of the images 1602 ₁ to 1602_(N) can be determined based on the approximate max marginal measures1902 (block 1834). Then, a most likely sequence of hypotheses 1612 aredetermined across the time-sequential images 1602 ₁ to 1602 _(N) (block1836). In one embodiment, the most likely sequence of hypotheses 1612are represented as marginal distributions 1904 over number of cells ateach of the images 1602 ₁ to 1602 _(N). These marginal distributions1904 over number of cells can be determined based on the approximatejoint distribution (block 1838), or in any other suitable manner. Theselected hypotheses 1612 are associated with characteristics 1608 of thecells 1600, including the estimated number 1906 of the cells 1600 _(i/n)shown in each of the images 1602 ₁ to 1602 _(N) (N=400 in the examplesshown in FIGS. 19A and 19B) and the geometry of the one or more cells1600 associated with each selected hypothesis 1612, as shown for each ofthe images 1602 ₁, 1602 ₁₅₀, 1602 ₂₅₀, and 1602 ₄₀₀, respectively. Theestimated number 1906 of the cells 1600 _(i/n) shown in each of theimages 1602 ₁ to 1602 _(N) can be determined based on crossover pointsbetween the marginal distributions 1904 for 1 cell (1904A), 2 cells(1904B), 3 cells (1904C), and 4 cells (1904D). The value of the marginaldistributions (y-axis) is plotted against image frame number (1 to 400).The value of each marginal distribution 1904 across the images 1602,represents the probability that the number of cells associated with themarginal distribution 1904 is shown in the images 1602 _(i), based onthe selected hypotheses 1612. The value of the estimated number 1906 ofthe cells 1600 _(i/n) (y-axis) is also plotted against image framenumber (1 to 400).

Example 3 below illustrates in detail examples of the mathematical formsof the approximate max marginal measures 1602 (see Eqs. (13) and (14),Example 3) and the approximate joint distribution over number of cellsat each of the images 1602 ₁ to 1602 _(N) (see Eq. (15), Example 3), andthat discussion is not repeated here. Eq. (15), Example 3 is generalizedto include information associated with tracking-free approaches such asclassification and/or interframe similarity determination that may beused in conjunction with cell activity tracking. When cell activitytracking is used without tracking-free approaches, an example of theapproximate joint distribution over number of cells is shown below asEq. (6), with the term c_(N) ^((t))(N^((t))) associated with theclassifier set to zero and the term δ^((t)) associated with interframesimilarity set to 0.5 (so that a division event is equally likely orunlikely between adjacent images 1602 (see FIG. 16A)).φ₂(d^((t)),δ^((t))) thereby becomes a constant and can be dropped fromthe approximate joint distribution.{circumflex over (P)}(N ^((t)))∝Π_(t=2) ^(t)({circumflex over (φ)}_(M)(N^((t)))ψ₂(N ^((t-1:t)) ,d ^((t)))  (6)

With reference to blocks 1836 and 1838, the marginal distributions 1904over number of cells at each of the images 1602 ₁ to 1602 _(N) can bedetermined using belief propagation. Belief propagation can be used tointegrate prior knowledge, enforce constraints (such as a non-decreasingnumber of cells), and fuse information such as cell tracking results,classification probabilities, and temporal image similarity to generateembryo stage estimates (such as the estimated number 1906 of the cells1600 _(i/n) shown in each of the images 1602 ₁ to 1602 _(N)) within aglobal context. In one embodiment, sum product belief propagation can beused to provide the joint distribution over number of cells at each ofthe images 1602 ₁ to 1602 _(N), and the marginal distributions 1904 overnumber of cells at each of the images 1602 ₁ to 1602 _(N). This set ofdistributions can be used to determine a confidence measure for theinferred cell division times (see description with reference to FIG.19C).

In one embodiment, the constraint taken into account by hypothesisselection (block 1836) is one of: (1) the inferred number of the one ormore cells 1600 associated with the hypotheses 1606 is non-decreasingwith time across the series of time-sequential images 1602 ₁ to 1602_(N); (2) after a change in the inferred number of the one or more cells1600, the inferred number of the one or more cells 1600 is stable for aperiod of time across a first subset of the series of time-sequentialimages 1602 ₁ to 1602 _(N); and/or (3) the inferred number of the one ormore cells 1600 decreases by no more than one with time across a secondsubset of the series of time-sequential images 1602 ₁ to 1602 _(N), thenincreases at the end of the second subset. Constraint (2) can facilitateelimination of some hypotheses 1606, such as cell division events thatoccur outside of expected biological timeframes. Constraint (3) canapply to human embryo development scenarios in which one or more of thecells 1600 divide, then recombine, then divide again later.

In one embodiment, the approximate inference over the PGM 1702 and/or1722 (see FIGS. 17A and 17B) described above may occur in a left toright fashion (from image 1602 ₁ to image 1602 _(N)) followed by eventinference (described with reference to FIG. 18C). Alternatively or inaddition, another pass through the images 1602 from right to left (fromimage 1602 _(N) to image 1602 ₁) can occur to further refine thehypotheses 1815 and to search for additional, as yet unexploredhypotheses. Alternatively or in addition, one or more passes through oneor more subsets of the images 1602 may occur.

In one embodiment, event inference (described with reference to FIG.18C) may be omitted. In this embodiment, the parent hypotheses 1817 (seeFIG. 18B) at each of the images 1602 ₁ to image 1602 _(N) may be theselected hypotheses 1612.

In the embodiment of FIG. 19A, cell activity tracking is used withouttracking-free approaches. Alternatively, in the embodiment of FIG. 19B,cell activity tracking is used in conjunction with classification andinterframe similarity determination. An image similarity measure 1905may relate to a likelihood of occurrence of one or more cell divisionevents between adjacent images 1602 (see FIG. 16A). A classificationmeasure 1903 may include estimated posterior probabilities of number ofcells (in one embodiment, in Eq. (8) (see Example 3) that may bedetermined from a set of hand-crafted and/or machine learneddiscriminative image features.

In some embodiments, the selected hypotheses 1612 associated with theplurality of images 1602 can be used to determine, account for, and/orotherwise be associated with characterization of biological activitybased on one or more parameters such as cell activity parameters, timingparameters, non-timing parameters, and/or the like. For example, whenthe plurality of images 1602 are time-lapse images of a developingembryo, each selected hypothesis 1612 can be associated with thelikelihood of the images 1602 showing a numbers of cells such as but notlimited to 1 cell, 2 cells, 3 cells, and/or 4 cells, and can be used toinfer cell division timing/events. In such embodiments, the selectedhypotheses 1612 can reflect constraints, such as those described withreference to FIG. 18C. Accordingly, the selected hypotheses 1612 can beused to determine, for the plurality of images 1602, duration of firstcytokinesis, a time interval between cytokinesis 1 and cytokinesis 2, atime interval between cytokinesis 2 and cytokinesis 3, a time intervalbetween a first and second mitosis, a time interval between a second andthird mitosis, a time interval from fertilization to an embryo havingfive cells (t5 in Table 2), a time interval from syngamy to the firstcytokinesis (S in Table 2), and/or other suitable parameters such asother parameters shown in Table 2.

In some embodiments, the parameters can include one or more parametersas described and/or referenced in Table 2, wherein the disclosure of(PCT Publication No.) WO 2012/163363, “Embryo Quality Assessment Basedon Blastomere Cleavage and Morphology,” International Filing Date May31, 2012 is incorporated by reference in its entirety.

Aspects of the invention are further operable for determination of aconfidence measure for each selected hypothesis (such as the hypotheses1612 (see FIG. 16A)). The confidence measure for each selectedhypothesis can be based on an estimate of the likelihood of the selectedhypothesis. If, for example, various periods in embryo development(including but not limited to 1 cell, 2 cell, 3 cell, and 4 cellperiods) are represented by marginal probabilities close to 1, andoptionally sharp transitions in the marginal distributions 1904 betweenthe 1 cell, 2 cell, 3 cell, and/or 4 cell regions, then the estimatednumber 1906 of the cells 1600 _(i/n) associated with the selectedhypotheses 1612 can be considered to be reliable with high confidence.The confidence measure can be expressed in any suitable manner, such asa probability (between 0 and 1), a percentage (between 0% and 100%),and/or the like.

In this manner, aspects of the invention are further operable todetermine if the selected hypothesis is reliable based on the confidencemeasure. For example, the selected hypothesis can be deemed reliable ifthe confidence measure meets or surpasses a threshold value, and deemedunreliable otherwise. In other words, the reliability determination canbe a binary selection criterion, and can be used to determine,automatically or manually, whether to use or discard the hypothesis,and/or the image associated with the hypothesis, and/or the plurality ofimages, and so on. In some embodiments, the reliability determinationcan be a factor affecting determination and/or communication of cellactivity parameters associated with the selected hypotheses. Forexample, in some embodiments, the cell activity parameters can bedetermined if at least one of the selected hypotheses for each differentcell characteristic is reliable. Hence, for example, cell activityparameters will be determined for the characteristics 108 (see FIGS.16A, 16B) if at least one selected hypothesis 1612 associated with eachof 1 cell, 2 cells, 3 cells, and 4 cells is deemed reliable. In someembodiments, the cell activity parameters can be determined if at leasta minimum number of selected hypotheses are reliable.

In some embodiments, the cell activity parameters are displayed only ifat least one of the selected hypotheses for each different number ofcells (e.g. for 1 cell, 2 cells, etc.) is deemed reliable. In someembodiments, the cell activity parameters are displayed with anindicator of the reliability of the selected hypotheses associatedtherewith. In this manner, aspects of the invention are operable toprevent display of low confidence results to a user, and/or to warn theuser of low reliability results.

In some embodiments, a selection criterion can be applied to the cellsshown in the plurality of images based on the reliability determinationof the images. In other words, the image-based reliability determinationcan be translated to making biological determinations of the cells shownin the images. For example, the selection criterion can be associatedwith development competence of the cells, i.e., whether the cells ifimplanted would proceed to blastocyst, would result in implantation in afemale subject, would result in a pregnancy when implanted in a femalesubject, and/or the like. In some embodiments, the one or more cells canbe deemed (for example) unfit for implantation if at least one of thehypotheses is determined to be unreliable. In some embodiments, theresult of applying such a selection criterion can be communicated to theuser. In this manner, the user can decide whether to discard or use thecells based on the image-based selection criterion determinationdescribed here.

FIG. 19C illustrates an exemplary and nonlimiting approach fordetermination of a confidence measure for selected hypotheses (such asselected hypotheses 1612 of FIG. 16A) and for applying this confidenceinformation, according to an embodiment of the invention. The estimatednumber 1906 of the cells 1600 _(i/n) shown in each of the images 1602 ₁to 1602 _(N) are associated with the selected hypothesis 1612 (for 1cell, 2 cells, 3 cells, or 4 cells, in this example) that has thehighest likelihood at each image 1602 _(i). Then, a confidence measurefor each selected hypothesis is determined (block 1908). The confidencemeasure can be representative of the reliability of one or more of theselected hypotheses 1612 across the images 1602 ₁ to 1602 _(N). Forexample, the confidence measure may be based on the highest probabilityassociated with the marginal distribution 1904B (for 2 cells; see FIGS.19A and 19B), or another suitable measure. Alternatively or in addition,the confidence measure may be based on sharpness of transitions betweenthe 1 cell, 2 cell, 3 cell, and/or 4 cell regions as represented by themarginal distributions 1904. If these various periods in embryodevelopment (including but not limited to 1 cell, 2 cell, 3 cell, and 4cell periods) are represented by marginal probabilities close to 1, andoptionally sharp transitions in the marginal distributions 1904 betweenthe 1 cell, 2 cell, 3 cell, and/or 4 cell regions, then the estimatednumber 1906 of the cells 1600 _(i/n) associated with the selectedhypotheses 1612 can be considered to be reliable with high confidence.The confidence measure may be a value between 0 and 1, and may representa percentage confidence value between 0% and 100%.

Next, the reliability of the selected hypotheses 1612 can be determinedby thresholding the confidence measure (block 1910). For example, theselected hypotheses 1612 can be deemed reliable overall if theconfidence measure for at least one selected hypothesis 1612 for eachnumber of cells is at least a threshold value. The threshold value maybe any suitable value between 0 and 1, such as but not limited to 0.5,0.6, 0.7, 0.8, 0.9, or 1.0. In some embodiments, if the selectedhypotheses 1612 are deemed unreliable, an indicator of the unreliabilityof the hypotheses 1612 may be displayed.

Next, if the selected hypotheses 1612 are deemed reliable, and/or if sospecified for unreliable outcomes, cell activity can be determined basedon characterization of parameters such as cell division events, durationof cell division and/or growth, and/or the like (block 1912). Next, aselection criterion can be applied to determine whether to accept orreject the embryo shown in the images 1602 for implantation (block1914). The selection criterion can be determined based on thethresholding performed at block 1910, and optionally based on theparameter characterization performed at block 1912.

In one embodiment, a rejection of an embryo for implantation into afemale human subject can be displayed if at least one of the hypotheses1612 is determined to be unreliable based on the selection criterion.Alternatively or in addition, an indicator of development competence ofthe embryo for implantation into a female human subject can bedisplayed, where the indicator is based on the reliability of at leastone of the hypotheses 1612 determined based on the selection criterion.The rejection and/or the indicator of development competence may bedisplayed along with an indicator of the reliability of the at least oneof the hypotheses 1612 based on the confidence measure.

Referring to FIG. 8, the system 800 is now described for automated celltracking and for confidence estimation in accordance with embodiments ofthe invention. In some embodiments, the display device 806 is furtherconfigured to present an indicator of the reliability of the pluralityof hypotheses.

In some embodiments, the computing apparatus 804 may be configured forautomated evaluation of cell activity. In some embodiments, thecomputing apparatus 804 may be configured to generate a plurality ofhypotheses characterizing one or more cells shown in an image, such thatthe plurality of hypotheses include an inferred characteristic of one ormore of the cells based on geometric features of the one or more cellsshown in the image. The computing apparatus may be further configured toselect a hypothesis from the plurality of hypotheses associated with theimage. The computing apparatus 804 may be further configured todetermine a characteristic of the one or more of the cells based on theinferred characteristic associated with the hypothesis. The one or morecells may be included in a multi-cell embryo. The one or more cells maybe included in a human embryo, one or more oocytes, or one or morepluripotent cells.

In some embodiments, the computing apparatus 804 may be configured toselect the hypothesis based on compatibility of the inferredcharacteristic with the geometric features of the one or more cellsshown in the image. The geometric features may include boundaryinformation associated with each of the one or more cells. The boundaryinformation may include one or more boundary segments. The computingapparatus may be configured to determine the one or more boundarysegments associated with each of the one or more cells.

In some embodiments, the computing apparatus 804 may be configured tomap a representation of each of the one or more cells to the one or moreboundary segments. In some embodiments, the computing apparatus 804 maybe further configured to map a first boundary segment to a nullidentifier associated with none of the cells, the boundary segmentsincluding the associated one or more of the boundary segments mapped tothe each of the cells and the first boundary segment.

In some embodiments, the computing apparatus 804 may be configured todetermine, based on the characteristic of the one or more of the cells,one or more of the following: a duration of first cytokinesis, a timeinterval between cytokinesis 1 and cytokinesis 2, a time intervalbetween cytokinesis 2 and cytokinesis 3, a time interval between a firstand second mitosis, a time interval between a second and third mitosis,a time interval from fertilization to an embryo having five cells, and atime interval from syngamy to the first cytokinesis.

In some embodiments, the computing apparatus 804 may be configured togenerate a preliminary hypothesis characterizing the one or more cellsshown in the image. The computing apparatus 804 may be furtherconfigured to refine the preliminary hypothesis to obtain one or more ofthe plurality of hypotheses based on the associated geometric featuresof the one or more cells shown in the image. The preliminary hypothesismay be refined based on a mapping of a representation of each of the oneor more cells to one or more boundary segments associated with each ofthe one or more cells.

In some embodiments, the preliminary hypothesis may include a pluralityof first shapes, each of the plurality of first shapes being defined byfirst shape parameter values, the each of the cells being characterizedby an associated one of the plurality of first shapes. The computingapparatus being configured to refine the preliminary hypothesis includesbeing configured to fit each of a plurality of second shapes to theassociated geometric features of the one or more cells shown in theimage. Each of the plurality of first shapes and each of the pluralityof second shapes may be ellipses. Alternatively, each of the pluralityof first shapes and each of the plurality of second shapes may beb-splines.

In some embodiments, the computing apparatus 804 may be configured todetermine boundary information associated with each of the one or morecells from a series of time-sequential images of the cells, the imagebeing a first image included in the series of time-sequential images.The computing apparatus 804 may be further configured to generate thepreliminary hypothesis by modifying a previously selected hypothesis,the previously selected hypothesis characterizing the cells as shown ina second image included in the series of time-sequential images, thesecond image prior to the first image. The series of time-sequentialimages may be a series of time-lapse images.

In some embodiments, the image may be a first image, and the computingapparatus 504 being configured to select the hypothesis from theplurality of hypotheses characterizing the cells as shown in the firstimage may include being configured to determine a most likely sequenceof hypotheses across a series of images including the first image.

In some embodiments, the series of images may be a series oftime-sequential images. The computing apparatus 804 being configured todetermine the most likely sequence of hypotheses across the series oftime-sequential images may include being configured to take into accounta constraint that limits how the inferred characteristic of the one ormore of the cells can vary across two or more of the series oftime-sequential images. The constraint may be selected from the groupconsisting of: (1) the inferred number of the one or more cells isnon-decreasing with time across the series of time-sequential images;(2) after a change in the inferred number of the one or more cells, theinferred number of the one or more cells is stable for a period of timeacross a first subset of the series of time-sequential images; and (3)the inferred number of the one or more cells decreases by no more thanone with time across a second subset of the series of time-sequentialimages, then increases at the end of the second subset.

In some embodiments, the inferred characteristic of the one or morecells may include at least one of an inferred number of the one or morecells and an inferred geometry of the one or more cells. Thecharacteristic of the one or more cells may include at least one of anumber of the one or more cells and a geometry of the one or more cells.The inferred geometry of the one or more cells may include an inferredshape of the one or more cells and an inferred arrangement of the one ormore cells. The geometry of the one or more cells may include a shape ofthe one or more cells and an arrangement of the one or more cells. Thenumber of the one or more cells may be the same as the inferred numberassociated with the hypothesis. The geometry of the one or more cellsmay be the same as the inferred geometry of the one or more cellsassociated with the hypothesis.

In some embodiments, the computing apparatus 804 may be configured toselect the hypothesis from the plurality of hypotheses based ondifferences between the inferred geometry of the one or more of thecells associated with each of the plurality of hypotheses and theassociated geometric features of the one or more cells shown in theimage. In some embodiments, the computing apparatus 804 may beconfigured to select the hypothesis from the plurality of hypothesesbased on compatibility between the inferred geometry of the one or moreof the cells associated with each of the plurality of hypotheses and theassociated geometric features of the one or more cells shown in theimage.

In some embodiments, the computing apparatus 804 may be configured todetermine the one or more boundary segments associated with each of theone or more cells. In some embodiments, the computing apparatus beingconfigured to determine the one or more boundary segments of each of theone or more cells may include being configured to perform segmentgeneration, such as but not limited to ridge search segment generation.In some embodiments, the computing apparatus 804 being configured todetermine the one or more boundary segments of each of the one or morecells may include being configured to merge a first boundary segment anda second boundary segment into a third boundary segment included in theone or more boundary segments of at least one of the one or more cells.

In some embodiments, the computing apparatus 804 may be configured todetermine a confidence measure associated with a plurality of hypothesesbased on an estimate of a likelihood of the one or more of the pluralityof hypotheses. Each of the plurality of hypotheses characterizes one ormore cells shown in an associated one or more of a plurality of images.In some embodiments, the computing apparatus 804 is further configuredto select the plurality of hypotheses based on differences between aninferred geometry of each of the one or more cells associated with eachof the plurality of hypotheses and boundaries of the each of the one ormore cells determined from the one or more images of the one or morecells. In some embodiments, the computing apparatus 804 is furtherconfigured to select the plurality of hypotheses based on compatibilitybetween an inferred geometry of each of the one or more cells associatedwith each of the plurality of hypotheses and boundaries of the each ofthe one or more cells determined from the one or more images of the oneor more cells.

In some embodiments, the plurality of images are a series of time-lapseimages, and the estimate of the likelihood of the one or more of theplurality of hypotheses takes into account a constraint that the numberof cells shown in each of the series of time-lapse images isnon-decreasing with time.

The computing apparatus 804 may be further configured to determinereliability of the plurality of hypotheses based on the confidencemeasure. In some embodiments, each of the plurality of hypotheses arebased on one or more of an estimate of a number of the one or morecells, an estimate of a shape of each of the one or more cells, and anestimate of an arrangement of the one or more cells.

In some embodiments, the computing apparatus 804 may be furtherconfigured to detect boundaries associated with the one or more cells ineach of the plurality of images. Each of the plurality of hypotheses maybe based on an associated one or more of the boundaries. In someembodiments, each of the boundaries includes one or more boundarysegments.

In some embodiments, the plurality of hypotheses are associated with acharacterization of cell activity associated with development potentialof the one or more cells. In some embodiments, the characterization ofcell activity includes one or more of the following: a duration of firstcytokinesis, a time interval between cytokinesis 1 and cytokinesis 2, atime interval between cytokinesis 2 and cytokinesis 3, a time intervalbetween a first and second mitosis, a time interval between a second andthird mitosis, a time interval from fertilization to an embryo havingfive cells, and a time interval from syngamy to the first cytokinesis.

In some embodiments, the display device 806 may be configured to displayan indicator of development competence of the one or more of the cellsfor implantation into a female human subject based on the characteristicof the one or more of the cells.

In some embodiments, the display device 806 may be further configured topresent an indicator of the reliability of the plurality of hypotheses,and the input interface 808 may be further configured to receive, inresponse to the presenting via the display device 806, an inputindicating the development competence of the one or more cells. In someembodiments, the display device 806 is configured to display thecharacterization of cell activity only if the at least one of theplurality of hypotheses is determined to be reliable. In someembodiments, the display device is configured to display thecharacterization of cell activity and an indicator of the reliability ofthe at least one of the plurality of hypotheses associated with thecharacterization of cell activity.

In some embodiments, the computing apparatus 804 may be furtherconfigured to perform classification to augment determination of thecharacteristic of the one or more cells. Alternatively or in addition,the computing apparatus 804 may be further configured to perform imagesimilarity determination to augment determination the characteristic ofthe one or more cells.

In some embodiments, the computing apparatus 804 may be furtherconfigured to apply a selection criterion to the one or more cells basedon the confidence measure as part of determining the reliability of theat least one of the plurality of hypotheses. In some embodiments, theselection criterion is associated with development competence of the oneor more cells for implantation into a female human subject. In someembodiments, the selection criterion is based on one or more thresholdvalues of the confidence measure. In some embodiments, the displaydevice 806 is configured to display a result of applying the selectioncriterion.

In some embodiments, the computing apparatus 804 may be furtherconfigured to reject the one or more cells for implantation into afemale human subject if the at least one of the plurality of hypothesesis determined to be unreliable based on the selection criterion. In someembodiments, the display device 806 may be further configured to displayan indicator of development competence of the one or more cells forimplantation into a female human subject based on the reliability of theat least one of the plurality of hypotheses determined based on theselection criterion.

Now referring to FIG. 9, in some embodiments, the processor 912 furtherincludes a boundary detection module 922, a hypothesis generation module924, a hypothesis selection module 926, a confidence module 928, areliability determination module 930, a mapping module 932, and a cellactivity parameter determination module 933.

In some embodiments, the hypothesis selection module 926 may beconfigured to select a hypothesis from a plurality of hypothesescharacterizing one or more cells shown in an image. Each of theplurality of hypotheses may include an inferred characteristic of one ormore of the cells based on geometric features of the one or more cellsshown in the image. The hypothesis selection module 926 may be furtherconfigured to determine a characteristic of the one or more of the cellsbased on the inferred characteristic associated with the hypothesis. Thehypothesis selection module 926 may be implemented in at least one of amemory or a processing device. The one or more cells may be included ina multi-cell embryo. The one or more cells may be included in a humanembryo, one or more oocytes, or one or more pluripotent cells.

In some embodiments, the hypothesis selection module 926 may beconfigured to select the hypothesis based on compatibility of theinferred characteristic with the geometric features of the one or morecells shown in the image. The geometric features may include boundaryinformation associated with each of the one or more cells. The boundaryinformation may include one or more boundary segments. The computingapparatus may be configured to determine the one or more boundarysegments associated with each of the one or more cells.

In some embodiments, the inferred characteristic of the one or morecells may include at least one of an inferred number of the one or morecells and an inferred geometry of the one or more cells. Thecharacteristic of the one or more cells may include at least one of anumber of the one or more cells and a geometry of the one or more cells.The number of the one or more cells may be the same as the inferrednumber associated with the hypothesis. The geometry of the one or morecells may be the same as the inferred geometry of the one or more cellsassociated with the hypothesis.

In some embodiments, the hypothesis selection module 926 may beconfigured to select the hypothesis from the plurality of hypothesesbased on differences between the inferred geometry of the one or more ofthe cells associated with each of the plurality of hypotheses and theassociated geometric features of the one or more cells shown in theimage. The hypothesis selection module 926 may be configured to selectthe hypothesis from the plurality of hypotheses based on compatibilitybetween the inferred geometry of the one or more of the cells associatedwith each of the plurality of hypotheses and the associated geometricfeatures of the one or more cells shown in the image.

In some embodiments, the image is a first image. The hypothesisselection module 926 may be configured to select the hypothesis from theplurality of hypotheses characterizing the cells as shown in the firstimage based on a determination of a most likely sequence of hypothesesacross a series of images including the first image.

In some embodiments, the series of images is a series of time-sequentialimages. The hypothesis selection module 926 may be configured todetermine the most likely sequence of hypotheses across the series oftime-sequential images taking into account a constraint limiting how theinferred characteristic of the one or more cells can vary across two ormore of the series of time-sequential images. The constraint may beselected from the group consisting of: (1) the inferred number of theone or more cells is non-decreasing with time across the series oftime-sequential images; (2) after a change in the inferred number of theone or more cells, the inferred number of the one or more cells isstable for a period of time across a first subset of the series oftime-sequential images; and (3) the inferred number of the one or morecells decreases by no more than one with time across a second subset ofthe series of time-sequential images, then increases at the end of thesecond subset.

In some embodiments, the hypothesis generation module 924 may beconfigured to generate the plurality of hypotheses based on theassociated geometric features of the one or more cells shown in theimage. The hypothesis generation module 924 may be configured togenerate a preliminary hypothesis characterizing the cells as shown inthe image, and may be configured to refine the preliminary hypothesis toobtain one or more of the plurality of hypotheses, based on thegeometric features of the one or more cells shown in the image. Thehypothesis generation module 924 may be configured to refine thepreliminary hypothesis based on a mapping of a representation of the oneor more cells to one or more boundary segments as characterized by thepreliminary hypothesis.

In some embodiments, the preliminary hypothesis includes a plurality offirst shapes, each of the plurality of first shapes being defined byfirst shape parameter values, each of the one or more of the cells beingcharacterized by an associated one of the plurality of first shapes. Thehypothesis generation module 924 may be configured to refine thepreliminary hypothesis based on a fit of each of a plurality of secondshapes to the associated geometric features of the one or more cellsshown in the image. Each of the plurality of first shapes and each ofthe plurality of second shapes may be ellipses. Alternatively, each ofthe plurality of first shapes and each of the plurality of second shapesmay be b-splines.

In some embodiments, the boundary detection module 922 may be configuredto determine boundary information associated with each of the one ormore cells based on the image. The boundary detection module 922 may befurther configured to determine the boundary information from a seriesof time-sequential images of the cells. The image may be a first imageincluded in the series of time-sequential images. The hypothesisgeneration module 924 may be further configured to determine thepreliminary hypothesis by modifying a previously selected hypothesis,the previously selected hypothesis characterizing the cells as shown ina second image included in the series of time-sequential images, thesecond image prior to the first image.

In some embodiments, the boundary detection module 922 may be configuredto determine the one or more boundary segments associated with each ofthe one or more of the cells based on the image. The boundary detectionmodule 922 may be further configured to perform segment generation, suchas but not limited to ridge search segment generation to determine theone or more boundary segments.

In some embodiments, the boundary detection module 922 may be configuredto determine the one or more boundary segments associated with each ofthe one or more of the cells based on the image. The boundary detectionmodule 922 may be further configured to perform segment merging todetermine at least one of the one or more boundary segments. Forexample, the boundary detection module 922 may be configured to merge afirst boundary segment and a second boundary segment into a thirdboundary segment included in the one or more boundary segments.

In some embodiments, the cell activity parameter determination module933 may be configured to determine, based on the characteristic of theone or more cells, one or more of the following: a duration of firstcytokinesis, a time interval between cytokinesis 1 and cytokinesis 2, atime interval between cytokinesis 2 and cytokinesis 3, a time intervalbetween a first and second mitosis, a time interval between a second andthird mitosis, a time interval from fertilization to an embryo havingfive cells, and a time interval from syngamy to the first cytokinesis.

In some embodiments, the mapping module 932 may be configured to map arepresentation of each of the one or more of the cells to the associatedone or more boundary segments as characterized by each of the pluralityof hypotheses. In some embodiments, the boundary segments may includethe one or more boundary segments and a first boundary segment. Themapping module may be configured to map the first boundary segment to anull identifier associated with none of the cells.

In some embodiments, the confidence module 928 may be configured todetermine a confidence measure associated with a plurality of hypothesesbased on an estimate of a likelihood of one or more of the plurality ofhypotheses. Each of the plurality of hypotheses characterizing one ormore cells shown in an associated one or more of the plurality ofimages.

In some embodiments, the reliability determination module 930 may beconfigured to determine reliability of at least one of the plurality ofhypotheses based on the confidence measure. In some embodiments, thereliability determination module 930 may be further configured to applya selection criterion to the one or more cells based on the confidencemeasure. In some embodiments, the selection criterion is associated withdevelopment competence of the one or more cells for implantation into afemale human subject.

In some embodiments, the plurality of hypotheses is a first plurality ofhypotheses, and the hypothesis generation module 924 may be configuredto determine a second plurality of hypotheses including the firstplurality of hypotheses. Each of the second plurality of hypotheses isbased on one or more of an estimate of a number of the one or morecells, an estimate of a shape of each of the one or more cells, and anestimate of an arrangement of the one or more cells.

In some embodiments, the hypothesis selection module 926 may beconfigured to select the plurality of hypotheses based on differencesbetween an inferred geometry of each of the one or more cells associatedwith each of the plurality of hypotheses and boundaries of the each ofthe one or more cells determined from the one or more images of the oneor more cells. In some embodiments, each of the boundaries includes oneor more boundary segments. In some embodiments, the hypothesis selectionmodule 926 may be configured to select the plurality of hypotheses basedon compatibility between an inferred geometry of each of the one or morecells associated with each of the plurality of hypotheses and boundariesof the each of the one or more cells determined from the one or moreimages of the one or more cells. In some embodiments, each of theboundaries includes one or more boundary segments.

In some embodiments, the boundary detection module 922 may be configuredto detect boundaries associated with the one or more cells in each ofthe plurality of images. Each of the plurality of hypotheses is based onan associated one or more of the boundaries. In some embodiments, eachof the boundaries includes one or more boundary segments.

In some embodiments, the display module 942 may be configured to displaythe characterization of cell activity only if the at least one of theplurality of hypotheses is determined to be reliable. In someembodiments, the display module 942 may be further configured to displaythe characterization of cell activity and an indicator of thereliability of the at least one of the plurality of hypothesesassociated with the characterization of cell activity. In someembodiments, the display module 942 may be further configured to displaya result of applying the selection criterion. In some embodiments, thedisplay module 942 may be further configured to display an indicator ofdevelopment competence of the one or more cells for implantation into afemale human subject based on the characteristic of the one or morecells, and/or based on the reliability of the at least one of theplurality of hypotheses determined based on the selection criterion.

In some embodiments, the classification module 936 may be configured toaugment determination of the characteristic of the one or more cells bythe hypothesis selection module 926.

In some embodiments, the processor 912 may further include an outcomedetermination module 938, which was previously described.

In some embodiments, the processor 912 may further include an imagesimilarity determination module 941. The image similarity determinationmodule 941 may be configured to augment determination of thecharacteristic of the one or more cells by the hypothesis selectionmodule 926.

FIG. 20 illustrates a method 2000 for automated evaluation of cellactivity, in accordance with an embodiment of the invention. A pluralityof hypotheses are generated characterizing the one or more cells (block2010). An inferred characteristic of the one or more cells may bedetermined based on geometric features of the one or more cells (block2012). Next, a hypothesis from the plurality of hypotheses is selected(block 2020). Next, a characteristic of the one or more of the cellsbased on the inferred characteristic associated with the firsthypothesis may be determined (block 2030).

In some embodiments, a method for automated, non-invasive evaluation ofcell activity, comprises generating a plurality of hypothesescharacterizing one or more cells shown in an image, the generating theplurality of hypotheses including determining an inferred characteristicof the one or more cells based on geometric features of the one or morecells shown in the image. The method for automated, non-invasiveevaluation of cell activity further includes selecting a hypothesis fromthe plurality of hypotheses associated with the image. The method mayinclude determining a characteristic of the one or more of the cellsbased on the inferred characteristic associated with the hypothesis.

In some embodiments, the one or more cells are included in a multi-cellembryo.

In some embodiments of the method for automated, non-invasive evaluationof cell activity, the selecting the hypothesis is based on compatibilityof the inferred characteristic with the geometric features of the one ormore cells shown in the image. The geometric features may includeboundary information associated with each of the one or more cells. Theboundary information may include one or more boundary segments.

In some embodiments, the method for automated, non-invasive evaluationof cell activity further includes mapping of a representation of each ofthe one or more cells to the one or more boundary segments. In someembodiments, the method for automated, non-invasive evaluation of cellactivity further includes mapping a first boundary segment to a nullidentifier associated with none of the cells, the boundary segmentsincluding the associated one or more of the boundary segments mapped tothe each of the one or more cells and the first boundary segment.

In some embodiments, the method for automated, non-invasive evaluationof cell activity further includes performing classification to augmentdetermination of the characteristic of the one or more cells.

In some embodiments, the method for automated, non-invasive evaluationof cell activity further includes performing image similaritydetermination to augment determination of the characteristic of the oneor more cells.

In some embodiments, the method for automated, non-invasive evaluationof cell activity further includes determining, based on thecharacteristic of the one or more cells, one or more of the following: aduration of first cytokinesis, a time interval between cytokinesis 1 andcytokinesis 2, a time interval between cytokinesis 2 and cytokinesis 3,a time interval between a first and second mitosis, a time intervalbetween a second and third mitosis, a time interval from fertilizationto an embryo having five cells, and a time interval from syngamy to thefirst cytokinesis.

In some embodiments, the inferred characteristic of the one or morecells includes at least one of an inferred number of the one or morecells and an inferred geometry of the one or more cells, and thecharacteristic of the one or more cells includes at least one of anumber of the one or more cells and a geometry of the one or more cells.In some embodiments, the inferred geometry of the one or more cellsincludes an inferred shape of the one or more cells and an inferredarrangement of the one or more cells. In some embodiments, the geometryof the one or more cells includes a shape of the one or more cells andan arrangement of the one or more cells. In some embodiments, the numberof the one or more cells is the same as the inferred number associatedwith the hypothesis. In some embodiments, the geometry of the one ormore cells is the same as the inferred geometry of the one or more cellsassociated with the hypothesis. In some embodiments, the selecting thehypothesis from the plurality of hypotheses is based on differencesbetween the inferred geometry of the one or more cells associated witheach of the plurality of hypotheses and the geometric features of theone or more cells shown in the image. In some embodiments, the selectingthe hypothesis from the plurality of hypotheses is based oncompatibility between the inferred geometry of the one or more cellsassociated with each of the plurality of hypotheses and the geometricfeatures of the one or more cells shown in the image.

In some embodiments, the method for automated, non-invasive evaluationof cell activity further includes displaying an indicator of developmentcompetence of the one or more cells for implantation into a female humansubject based on the characteristic of the one or more cells.

In some embodiments, the method for automated, non-invasive evaluationof cell activity further includes determining the one or more boundarysegments associated with each of the one or more cells. In someembodiments, determining the one or more boundary segments of each ofthe one or more cells includes performing segment generation, such asbut not limited to ridge search segment generation. In some embodiments,determining the one or more boundary segments of each of the one or morecells includes merging a first boundary segment and a second boundarysegment into a third boundary segment included in the one or moreboundary segments of at least one of the one or more cells.

In some embodiments, the method for automated, non-invasive evaluationof cell activity further includes generating a preliminary hypothesischaracterizing the one or more cells, and refining the preliminaryhypothesis to obtain one or more of the plurality of hypotheses based onthe associated geometric features of the one or more cells shown in theimage. In some embodiments, the preliminary hypothesis includes aplurality of first shapes, each of the plurality of first shapes beingdefined by first shape parameter values, the each of the cells beingcharacterized by an associated one of the plurality of first shapes. Insome embodiments, the refining the preliminary hypothesis includesfitting each of a plurality of second shapes to the associated geometricfeatures of the one or more cells shown in the image. In someembodiments, each of the plurality of first shapes and each of theplurality of second shapes are ellipses. In some embodiments, each ofthe plurality of first shapes and each of the plurality of second shapesare b-splines. In some embodiments, the method for automated,non-invasive evaluation of cell activity further includes determiningboundary information associated with each of the one or more cells froma series of time-sequential images of the cells, the image being a firstimage included in the series of time-sequential images, and generatingthe preliminary hypothesis by modifying a previously generatedhypothesis, the previously generated hypothesis characterizing the cellsas shown in a second image included in the series of time-sequentialimages, the second image prior to the first image. In some embodiments,the series of time-sequential images is a series of time-lapse images.

In some embodiments, the image is a first image, and the selecting thehypothesis from the plurality of hypotheses characterizing the one ormore cells as shown in the first image includes determining a mostlikely sequence of hypotheses across a series of images including thefirst image. In some embodiments, the series of images is a series oftime-sequential images, and the determining the most likely sequence ofhypotheses across the series of time-sequential images takes intoaccount a constraint that limits how the inferred characteristic of theone or more cells can vary across two or more of the series oftime-sequential images. In some embodiments, the constraint is selectedfrom the group consisting of: an inferred number of the one or morecells is non-decreasing with time across the series of time-sequentialimages; after a change in the inferred number of the one or more cells,the inferred number of the one or more cells is stable for a period oftime across a first subset of the series of time-sequential images; andthe inferred number of the one or more cells decreases by no more thanone with time across a second subset of the series of time-sequentialimages, then increases at the end of the second subset.

In some embodiments, the cells are included in a human embryo, one ormore oocytes, or one or more pluripotent cells.

FIG. 21 illustrates a method 2100 of the invention for automatedevaluation of cell activity including reliability determination, inaccordance with an embodiment of the invention. A confidence measure isdetermined, the confidence measure associated with a plurality ofhypotheses based on an estimate of a likelihood of one or more of theplurality of hypotheses (block 2110). Each of the plurality ofhypotheses characterizes one or more cells shown in an associated one ormore of a plurality of images. Reliability of at least one of theplurality of hypotheses is determined based on the confidence measure(block 2120).

In some embodiments, a method for automated evaluation of cell activitycomprises determining a confidence measure associated with a pluralityof hypotheses based on an estimate of a likelihood of one or more of theplurality of hypotheses, each of the plurality of hypothesescharacterizing one or more cells shown in an associated one or more of aplurality of images. The method for automated evaluation of cellactivity further includes determining reliability of at least one of theplurality of hypotheses based on the confidence measure.

In some embodiments, the one or more cells are included in a humanembryo, one or more oocytes, or one or more pluripotent cells.

In some embodiments, the method for automated evaluation of cellactivity further includes electing the plurality of hypotheses based ondifferences between an inferred geometry of each of the one or morecells associated with each of the plurality of hypotheses and boundariesof the each of the one or more cells determined from the one or moreimages of the one or more cells. In some embodiments, each of theboundaries includes one or more boundary segments.

In some embodiments, the plurality of images are acquired by dark-fieldillumination microscopy.

In some embodiments, each of the plurality of hypotheses are based onone or more of an estimate of a number of the one or more cells, anestimate of a shape of each of the one or more cells, and an estimate ofan arrangement of the one or more cells.

In some embodiments, the method for automated evaluation of cellactivity further includes detecting boundaries associated with the oneor more cells in each of the plurality of images, wherein each of theplurality of hypotheses is based on an associated one or more of theboundaries. In some embodiments, each of the boundaries includes one ormore boundary segments.

In some embodiments, the plurality of hypotheses are associated with acharacterization of cell activity associated with development potentialof the one or more cells. In some embodiments, the characterization ofcell activity includes one or more of the following: a duration of firstcytokinesis, a time interval between cytokinesis 1 and cytokinesis 2, atime interval between cytokinesis 2 and cytokinesis 3, a time intervalbetween a first and second mitosis, a time interval between a second andthird mitosis, a time interval from fertilization to an embryo havingfive cells, and a time interval from syngamy to the first cytokinesis.In some embodiments, the method for automated evaluation of cellactivity further includes displaying the characterization of cellactivity only if the at least one of the plurality of hypotheses isdetermined to be reliable. In some embodiments, the method for automatedevaluation of cell activity further includes displaying thecharacterization of cell activity and an indicator of the reliability ofthe at least one of the plurality of hypotheses associated with thecharacterization of cell activity.

In some embodiments, the plurality of images are a series of time-lapseimages, and the estimate of the likelihood of the one or more of theplurality of hypotheses takes into account a constraint that the numberof cells shown in each of the series of time-lapse images isnon-decreasing with time.

In some embodiments, determining the reliability of the at least one ofthe plurality of hypotheses includes applying a selection criterion tothe one or more cells based on the confidence measure. In someembodiments, the selection criterion is associated with developmentcompetence of the one or more cells for implantation into a female humansubject. In some embodiments, the selection criterion is based on one ormore threshold values of the confidence measure. In some embodiments,the method for automated evaluation of cell activity further includesdisplaying a result of applying the selection criterion. In someembodiments, the method for automated evaluation of cell activityfurther includes rejecting the one or more cells for implantation into afemale human subject if the at least one of the plurality of hypothesesis determined to be unreliable based on the selection criterion. In someembodiments, the method for automated evaluation of cell activityfurther includes displaying an indicator of development competence ofthe one or more cells for implantation into a female human subject basedon the reliability of the at least one of the plurality of hypothesesdetermined based on the selection criterion.

Example 3

Human embryo tracking can face challenges including a high dimensionalsearch space, weak features, outliers, occlusions, missing data,multiple interacting deformable targets, changing topology, and a weakmotion model. This example address these by using a rich set ofdiscriminative image and geometric features with their spatial andtemporal context. In one embodiment, the problem is posed as augmentedsimultaneous segmentation and classification in a conditional randomfield (CRF) framework that combines tracking based and tracking freeapproaches. A multi pass data driven approximate inference on the CRF isperformed. Division events were measured during the first 48 hours ofdevelopment to within 30 minutes in 65% of 389 clinical image sequences,winch represents a 19% improvement over a purely tracking based ortracking free approach.

Augmented Simultaneous Segmentation and Classification

Augmented simultaneous segmentation and classification leveragestracking based and tracking free approaches to estimate division events.Both types of features are extracted and added to a CRF. Approximateinference is then performed.

Feature Extraction

In one embodiment, the image features used for tracking are segments1604, depicted in FIG. 16A. Fewer segments 1604 reduces the number oftracks. In this example, boundary points are extracted using a Hessianoperator, which provides a strength and orientation angle for eachpixel. A directed local search is conducted for coherent boundary pixelsusing this information with hysteresis thresholding. A subsequentmerging inference combines the segments into a smaller set of largersegments. This step is formulated as a graph partitioning on a graphwhose vertices are segments and whose edges indicate merging ofsegments. The number of partitions is unknown in advance.

In one embodiment, the tracking free portion of the framework uses a perframe classifier trained on number of cells (such as the classifier 102described with reference to FIG. 1A), and an interframe similaritymeasure. In this example, the classifier uses a rich set of 262 handcrafted and automatically learned discriminative features. Thesimilarity measure may be a normalized cross correlation (NCC).

CRF Model

This example seeks to estimate the numbers and shapes of cells in theembryo over time, as depicted in FIG. 16A as characteristics 1608 of thecells 1600. A stochastic evolution of elliptical cells with the CRF inFIG. 17B is modeled. As previously described with reference to FIGS. 17Aand 17B, at each frame t there are K_(t) segments, each with m_(k)^((t)) points, kε{1 . . . K_(t)}, and up to N_(max) cells. The variablesto be inferred are labels assigning segments to cells l^((t))ε{0, 1, . .. N_(max)}^(K) ^(t) ; ellipses e_(n) ^((t))ε

⁵, nε{1, . . . N_(max)}; number of cells N^((t))ε{1, . . . N_(max)}; anddivision event d^((t))ε{0,1}. Each ellipse e_(n) ^((t)) is associatedwith its parent, e_(Pa(n)) ^((t-1)). The observations are segmentss^((t))={s_(k) ^((t))}_(k=1 . . . K) _(t) where s_(k) ^((t)) is acollection of points s_(k,i) ^((t))ε

² with iε{1 . . . m_(k) ^((t))}; a classifier on the number of cellsc_(N) ^((t))ε

^(N) ^(max) ; and image similarity measure δ^((t))ε[0,1]. Compatibilityfunctions are either over variables that: (1) are within one time slice(observation model Φ); or (2) span neighboring time slices (motion modelΨ). The CRF encodes the joint probability distribution over allvariables as proportional to the product of all compatibility functions:

$\begin{matrix}{{P\left( {e^{({1:T})},l^{({1:T})},N^{({1:T})},d^{({1:T})},s^{({1:T})}} \right)} = {\frac{1}{z_{T}}{\prod\limits_{t = 1}^{T}\;{\underset{\underset{{observation}\mspace{14mu}{model}}{︸}}{\Phi\left( {e^{(t)},l^{(t)},N^{(t)},s^{(t)}} \right)} \cdot {\prod\limits_{t = 2}^{T}\;\underset{\underset{{motion}\mspace{14mu}{model}}{︸}}{\psi\left( {e^{({{t - 1}:t})},N^{({{t - 1}:t})},d^{(t)}} \right)}}}}}} & (7)\end{matrix}$

where T is the sequence length and Z_(T) is a normalizing constant. Weare interested in the sequence N^((t)) which maximizes the marginaldistribution P(N^((t))).

The observation model Φ is the product of three compatibility functions:Φ(e ^((t)) ,l ^((t)) ,N ^((t)) ,d ^((t)) ,s ^((t)),δ^((t)) ,c _(N)^((t)))=φ₀(e ^((t)))(φ₁(e ^((t)) ,l ^((t)) ,N ^((t)) ,s ^((t)))c _(N)^((t))(N ^((t))))φ₂(d ^((t)),δ^((t)))  (8)

The function φ₀ encodes limits on shape. The second function combinesclassifier c_(N) ^((t)) with φ₁, which encodes compatibility ofellipses, segments, and labels,φ₁(e ^((t)) ,l ^((t)) ,N ^((t)) ,s ^((t)))(Π_(t=1) ^(N) ^((t))f(•,•,•)^(c) ^(f) e ^(−c) ^(r) ^(r(•,•,•)) ² )^((1/N) ^((t)) ⁾  (9)

where f(e_(i) ^((t)),l^((t)),s^((t)))ε[0,1] is an ellipse coverage term,r(e_(i) ^((t)),l^((t)),s^((t)))ε

₊ is segment fitting error, and c_(f) and c_(r) are empirically chosen.

The function φ₂ relates division with similarity δ^((t)) of adjacentimages.

$\begin{matrix}{{\phi_{2}\left( {d^{(t)},\delta^{(t)}} \right)} = \left\{ \begin{matrix}\delta^{(t)} & {{{if}\mspace{14mu} d^{(t)}} = 0} \\{1 - \delta^{(t)}} & {otherwise}\end{matrix} \right.} & (10)\end{matrix}$

The transition model Ψ governs cell shape deformations and division:

$\begin{matrix}{{\psi\left( {e^{({{t - 1}:t})},N^{({{t - 1}:t})},d^{(t)}} \right)}_{t = {2\mspace{14mu}\ldots\mspace{14mu} T}} = {\prod\limits_{n = 1}^{N^{(t)}}\;{{\psi_{1}\left( {e_{P_{a{(n)}}}^{({t - 1})},e_{n}^{(n)},d^{(t)}} \right)}{\psi_{2}\left( {N^{({{t - 1}:t})},d^{(t)}} \right)}}}} & (11)\end{matrix}$

The function Ψ₁ encodes the underlying cell deformation and divisionprocess,

$\begin{matrix}{{\psi_{1}\left( {e_{i_{1}}^{({t - 1})},e_{i_{2}}^{(t)},d^{(t)}} \right)} = \left\{ \begin{matrix}{\mathbb{e}}^{- {\rho{({e_{i_{1}}^{({t - 1})},e_{i_{2}}^{(t)}})}}} & {{{where}\mspace{14mu} d^{(t)}} = {{0\mspace{14mu}{or}\mspace{14mu} i_{1}} \neq P_{a{(i_{2})}}}} \\{\mathbb{e}}^{- {\rho({h{({e_{i_{1}}^{({t - 1})},e_{i_{2}}^{(t)}})}}}} & {{{{where}\mspace{14mu} d^{(t)}} = 1},{i_{1} = P_{a{(i_{2})}}}} \\0 & {otherwise}\end{matrix} \right.} & (12)\end{matrix}$

where ρ(e_(i) ₁ , e_(i) ₂ )=(e_(i) ₁ −e_(i) ₂ )^(T)Λ(e_(i) ₁ −e_(i) ₂ )with Λ a diagonal matrix of deformation costs, and h a non-affinetransform from a mother to daughter cell shape.

In this example, the function Ψ₂ constrains the number of cells N^((t))to be nondecreasing.

Approximate Inference

This example seeks the most likely sequence N^((t)) from the CRF.Approximate inference is performed in three phases: cell countclassification, approximate max marginal inference and event inference.

Cell count classification is part of the tracking free portion. In thisexample, the cell count classification uses a multilevel AdaBoostclassifier to estimate posterior probabilities of number of cells (c_(N)^((t)) in Eq. (8)) from a rich set of 262 hand crafted and automaticallylearned discriminative image features.

In this example, the max marginal inference is tracking based, andinfers geometry from segments. It estimates {circumflex over(Φ)}_(M)(N^((t))), the unnormalized max marginal measure of N^((t)) byoptimizing to time t on a mutilated subgraph that excludes c_(N) ^((t))and δ^((t)):

${{{\hat{\Phi}}_{M}\left( N^{(t)} \right)}{\max\limits_{e,s,l,N^{({1:{t - 1}})}}{E(t)}}},{where}$$\begin{matrix}{{E(t)} = {\prod\limits_{\tau = 1}^{t}\;{{\phi_{o}\left( e^{(\tau)} \right)}{\phi_{1}\left( {e^{(\tau)},s^{(\tau)},l^{(\tau)},N^{(\tau)}} \right)}{\psi\left( {e^{({{\tau - 1}:\tau})},N^{({{\tau - 1}:\tau})}} \right)}}}} & {{~~~~~~~}(13)} \\{= {{E\left( {t - 1} \right)}{\phi_{0}\left( e^{(t)} \right)}{\phi_{1}\left( {e^{(t)},s^{(t)},l^{(t)},N^{(t)}} \right)}{\psi\left( {e^{({{t - 1}:t})},N^{({{t - 1}:t})}} \right)}}} & {(14)}\end{matrix}$

This example maximizes this recursion with data driven sequential MonteCarlo (DD-SMC). A data driven refinement stage between the time andmeasurement updates reduces the required number of particles by refiningan initial particle to the incomplete set of boundary points withexpectation maximization (EM). {circumflex over (Φ)}_(M)(N^((t))), isthen taken from the particles. Exemplary results for the approximate maxmarginal measures 402 (see Eqs. (13) and (14)) are shown in FIGS. 19Aand 19B. Exemplary results for the classification measure 403 and theimage similarity measure 1905 are shown in FIG. 19B.

The event inference combines {circumflex over (Φ)}_(M)(N^((t))) with theclassifier c_(N) ^((t)) and image similarity δ^((t)) to obtain theapproximate marginal distribution on number of cells {circumflex over(P)}(N^((t))). It is performed over another mutilated subgraphcontaining N^((t)), d^((t)), δ^((t)) and c_(N) ^((t)), and estimates themost likely sequence {circumflex over (N)}^((t)). This exampleapproximates this subgraph by a chain graph whose nodes are N(t), andwhose joint distribution is factorized by unary terms (Eq. (8)) andpairwise terms (Eqs. (10,11)):

${\hat{P}\left( N^{(t)} \right)} \propto {\prod\limits_{T = 2}^{t}\;{\left( {{{\hat{\Phi}}_{M}\left( N^{(t)} \right)} + {c_{N}^{(t)}\left( N^{(t)} \right)}} \right){\phi_{2}\left( {d^{(t)},\delta^{(t)}} \right)}{\psi_{2}\left( {N^{({{t - 1}:t})},d^{(t)}} \right)}}}$

This example performs belief propagation to find the marginaldistributions 1904 (see FIGS. 19A and 19B). The value of the estimatednumber 1906 of cells is plotted against image frame number (see FIGS.19A and 19B), where the transition times between different estimatednumbers 1906 of cells are based on the crossover points in the marginaldistributions 1904 for the different numbers of cells (in this example,1-cell, 2-cell, 3-cell, and 4-cell).

Experimental Results

This example applied the algorithm to human embryo image sequencesacquired from multiple IVF clinics and followed for at least 3 days.Images were acquired with a dark field digital microscope and cropped to151×151 pixels every 5 minutes.

Tracking Performance

The algorithm was trained on 327 embryos and tested on a separate set of389 embryos. The times of first, second, and third mitosis t₁, t₂, andt₃ respectively, were measured. Two expert embryologists measured groundtruth for evaluation. rmsd was measured: the rms deviation between thealgorithm's measurements and those of the two panelists.

FIG. 22 illustrates exemplary results for the ratio of embryos for whichthe deviation from panelists is within a margin m of the interpanelistdisagreement (rmsd<d_(p)+m) for each transition (t₁, t₂, t₃) and overall transitions, in accordance with an embodiment of the invention. Thisis shown for three combinations of observables: (a) classifierprobabilities and similarity measure (tracking free), (b) DD-SMC maxmarginals (tracking based), and (c) all observables (combined). It canbe seen that sometimes one approach works better than the others. Forexample in the t₃ transition (the most difficult to determine), trackingfree can outperform tracking based, as the scene is more complex and maynot be adequately modeled with simple shape and outlier assumptions. Bycontrast in the t₂ transition, shape and structure of the two cell casecan be modeled better by the tracking based shape model than by bag offeatures in the tracking free approach. But in all cases, combining thetwo approaches yields substantial improvement. The fraction of datasetsfor which transitions were measured with an rmsd within 30 minutes ofthe inter panelist variation are shown in Table 5 for each of theindividual transition times as well as for all transition times. On thisdataset, 65.3% of embryos were tracked on all three transitions with anrmsd within 30 minutes of the interpanelist variation using the combinedapproach. This result can be compared with the corresponding rmsd of thetracking free and tracking based approaches in isolation, which wererespectively 54.5% and 55.0%. The relative improvement over trackingfree and tracking based approaches are respectively 19.8% and 18.7%. Itshould be noted that over 21% of these cases had an interpanelistdisagreement of over 30 minutes.

This suggests that tracking based and tracking free approaches can becombined to achieve automated tracking on a significant portion ofclinical data.

TABLE 5 Fraction of datasets tracked to within 30 minutes rmsd ofpanelists for tracking free, tracking based, and combined approachesapproach t₁ t₂ t₃ all tracking free 0.899 0.624 0.606 0.545 trackingbased 0.897 0.678 0.542 0.550 combined 0.933 0.722 0.673 0.653

Conclusion

The framework presented in this example combines multiple features andtheir contexts in a unified CRF framework that leverages tracking basedand tracking free approaches. Automated tracking comparable to manualexpert measurements in 65% of the test data is demonstrated, and can befurther enhanced by leveraging and learning from more labeled data as itbecomes available, as well as expanding the inference to explore largerportions of the solution space.

Automated Embryo Ranking and/or Categorization

Aspects of the invention are further operable for determination of aranking and/or a categorization for each embryo included in a pluralityof embryos. The ranking and/or categorization of each embryo can bebased on a score associated with a developmental potential of eachembryo, such as the quality of each embryo for a positive fertilityoutcome such as a pregnancy after implantation of the embryo. The scoremay be determined based on various types of features, such as but notlimited to: shape, texture, and/or edge features (described withreference to Table 1); timing features (such as various timingparameters described with reference to Table 2); biological features(such as but not limited to various other parameters described withreference to Table 2, likelihood of a genetic defect, likelihood ofaneuploidy, and degree of fragmentation associated with each embryo);machine-learned features (described with reference to Table 1); andspatial-temporal features (such as but not limited to combinations oftemporal features and spatial features, as described below). Thefeatures may be associated with the embryo and/or individual cellswithin the embryo. The features may be image features that are based oncharacteristics of the embryo and/or individual cells within the embryoshown in one or more images, and/or additional properties of the one ormore images, such as but not limited to the image background intensity.In one embodiment, the features may be image features associated with aboundary of an embryo shown in the image such as cross-boundaryintensity profile, cross-boundary texture profile, boundary curvature,and boundary ruffling. The features may also include morphology andother features that may be evaluated by medical professionals, and thatmay be provided as inputs to automated ranking and/or categorization ofembryos. The score may also be based on patient information and/orclinic related information, as further described below. The score mayalso be determined based on the confidence measure associated with thetiming features, as described with reference to FIG. 19C. For example,one or more timing features may be included or excluded in determinationof the score based on whether the confidence measure described withreference to FIG. 19C is above or below a threshold.

In one embodiment, the score can assume any of a large number of values,such as at least 10, 20, 50, 100, 1000, or a number of values that ismany orders of magnitude larger than 1000 (effectively infinite). Thescore may be real-valued or integer-valued. The score may besubstantially continuous to the limits of precision of the devicecomputing the score. The score may be normalized to a range such as [0,1], though this is not required. A higher score may be associated with ahigher quality embryo, or alternatively a lower score may be associatedwith a higher quality embryo.

Advantageously, ranking and/or categorization of embryos based on ascore that can assume any of a large number of values allows for embryosof similar quality to be differentiated. For example, a patient may havemultiple embryos that would be associated with the same class by anoutcome-based classifier and/or a cell-based classifier (both describedabove). Alternatively or in addition, the patient may have multipleembryos that have similar characteristics determined through cellactivity tracking (also described above). If, for example, the classand/or the embryo characteristics are associated with good developmentalpotential of the embryo, it may be desirable to perform automatedranking and/or categorization of the embryos to facilitate selection ofa subset of the embryos for implantation based on the ranking and/orcategorization. As described above, the automated ranking and/orcategorization of the embryos may be based on a wide variety offeatures, and therefore may provide a more accurate ranking and/orcategorization of the embryos than, for example, a visual comparison ofmorphology of the embryos by itself. Also, it may be desirable to rankand/or categorize the embryos of the patient within a larger group ofembryos, such as the embryos of patients from a population such as asingle clinic, a group of clinics, or an overall population. In thisway, a more reliable measure of the actual quality of the embryos of anindividual patient can be obtained, which provides information beyondthat associated with a relative ranking of only the embryos of theindividual patient.

In addition, there are various situations in which categorizing embryosinstead of or in addition to ranking embryos may be beneficial. Forexample, there may be circumstances in which direct comparison of embryoscores to a high precision may not be reliable, such as comparisonbetween embryos from different clinics that may have been subject todifferent environmental conditions. For this reason among others, in oneembodiment, embryos may be associated with corresponding categories.These categories may be outcomes such as, for example, high qualityblast, blast, and arrested, and may be based on ranges of scoresassociated with each category. Alternatively or in addition, thesecategories may be based on ranges of scores associated with eachcategory that do not map directly to outcomes. In one example, thesecategories may be high, medium, and low. In another example, thesecategories may be associated with a likelihood of blastocyst, such as90% likely blast, 70% likely blast, 50% likely blast, 30% likely blast,and 10% likely blast. The ranges of scores associated with each categorymay be based on scores associated with particular percentile ranges in apopulation of embryos such as in a single clinic, a group of clinics, oran overall population. In one embodiment, embryos may be associated withcorresponding categories based on ranking of the embryos, such asranking of the embryos within embryos of a single patient and/or rankingwithin embryos of a population. Alternatively or in addition, the rangesof scores associated with each category may shift depending on theoverall range of scores associated with the embryos being categorized.For example, if the categories are implant, freeze, and discard, therange of scores associated with implant may shift so that at least oneembryo is recommended for implantation into a given patient.

FIG. 23 illustrates a non-limiting example of an automated embryoranking and/or embryo categorization approach applied to images 2308 ofembryos, in accordance with an embodiment of the invention. Duringtraining, N sets of training images 2302 are provided with at least oneof label information (such as associated embryo quality and/or outcome,as described with reference to FIG. 4) and feature information. Featureextraction 2304 is then carried out from each series of the trainingimages 2302. The extracted feature information and their associatedlabel information can be employed to perform training 2305 of a rankingand/or categorization model 2306. The ranking and/or categorizationmodel 2306 may be a classifier, a neural network, or any other suitablemodel configured to perform supervised learning and/or unsupervisedlearning, as described above with reference to Example 1. Featureextraction 2310 is also carried out from each series of the images 2308.A single series of the images 2308 is shown in FIG. 23, but it isunderstood that feature extraction 2310 and ranking and/orcategorization 2312 are performed on one or more series of images 2308.The ranking and/or categorization 2312 is based on the ranking and/orcategorization model 2306 after training, and is configured to determinea score associated with a developmental potential of embryos shown inthe images 2308, and to rank and/or categorize the embryos shown in theimages 2308 based on the score associated with each of the embryos.

In one embodiment, the ranking and/or categorization model 2306 canproduce an output that can assume any of a large number of values, suchas a substantially continuous output. The output may be real-valued, ormay be quantized or normalized to an integer value. The ranking and/orcategorization model 2306 may be a neural network or a classifier thatmay be Naïve Bayes, Adaboost, Support Vector Machine (SVM), Boosting,Random Forrest, or any other that can produce a substantially continuousprediction score.

In one embodiment, the images 2302 may be a time-sequential series ofimages, such as a video of embryo development. Temporal features may beobtained, for example, from images of embryos taken at different times.Alternatively or in addition, one or more images in the series of images2302 may have a first depth of field, and one or more other images inthe series of images 2302 may have a second, different depth of field.The depth of field can be adjusted to focus on embryo features at aparticular depth or range of depths. Alternatively or in addition, oneor more images in the series of images 2302 may have a first modality,and one or more other images in the series of images 2302 may have asecond, different modality. Modalities may be, for example, brightfieldfor facilitating visual inspection of embryos such as for morphologicalgrading, and darkfield for minimizing exposure of embryos to light.Alternatively or in addition, one or more images in the series of images2302 may have a first angular orientation relative to the embryo, andone or more other images in the series of images 2302 may have a second,different angular orientation relative to the embryo to show differentangular planes within embryo. Spatial features may be obtained, forexample, from images of embryos taken at approximately the same time butat different depths of field, at different angular orientations, and/orusing different modalities. Spatial-temporal features may be obtained,for example, from images of embryos taken at different times and atdifferent depths of field, at different angular orientations, and/orusing different modalities.

As illustrated in FIG. 23, feature information can be extracted from thetraining images 2302 and the images 2308 through feature extraction 2304and feature extraction 2310, respectively, using various previouslydescribed approaches. For example, feature information such ashandcrafted shape, texture, and/or edge features and such asmachine-learned features (such as BoF) can be extracted as describedwith reference to FIGS. 1-15 and Example 1 (describing image-based cellclassification and outcome-based classification). In addition, featureinformation such as cell activity related parameters such as timing andbiological parameters (such as parameters described with reference toTable 2) can be determined through cell activity tracking as describedwith reference to FIGS. 16-22 and Examples 2 and 3, throughclassification approaches described with reference to FIGS. 1-15 andExample 1, through image similarity determination as described withreference to FIG. 19B and Example 3, and/or through combinations of theabove approaches.

As illustrated in FIG. 23, the ranking and/or categorization model 2306can be trained as described with reference to FIGS. 1-15 and Example 1(describing image-based cell classification and outcome-basedclassification). Alternatively or in addition, the ranking and/orcategorization model 2306 can be trained in any suitable manner known toone of ordinary skill in the art so that the ranking and/orcategorization model 2306 can take into account the various featuretypes previously described.

In one embodiment, the score associated with the developmental potentialof each embryo may be determined based on patient information and/orclinic related information. The patient information may be associatedwith one or more of a male parent of each embryo, and a female parent ofeach embryo. The patient information includes one or more of thefollowing: age; ethnicity; other demographic information; reproductivehistory; assisted reproductive history; and for the male parent,motility of sperm provided by the male parent and other parametersassociated with the sperm. The clinic related information may includeenvironmental parameters associated with the developmental potential ofeach embryo, such as media type in which each embryo is grown, incubatortemperature, and gas mix used in the incubator.

In one embodiment, the score associated with the developmental potentialof each embryo may be determined based on one or more of the followingfactors: likelihood of each embryo reaching blastocyst stage; likelihoodof each embryo reaching blastocyst stage with a quality level;likelihood of pregnancy resulting from each embryo; likelihood ofimplantation resulting from each embryo; likelihood of a genetic defectin each embryo; likelihood of aneuploidy associated with each embryo;degree of fragmentation associated with each embryo; whether one or moreof the embryos will be frozen; whether one or more of the embryosresulted from intra-cytoplasmic sperm injection (ICSI); whether the oneor more of the first plurality of embryos resulted from traditionalin-vitro fertilization; and medications used in connection with in-vitrofertilization resulting in one or more of the embryos.

In one embodiment, the training 2305 of the ranking and/orcategorization model 2306 may be based on at least one of labelinformation and feature information associated with each series oftraining images 2302. Each series of images 2302 (such as series ofimages 2302-1 and 2302-N in FIG. 23) may have associated labelinformation such as: an indication of quality of the correspondingembryo (such as high or low) for a positive fertility outcome; anindication of one of a plurality of outcomes for development of thecorresponding embryo, such as blast and arrested; implantation and noimplantation; pregnancy and no pregnancy; or ploidy and aneuploidy.Outcomes may include one or more sets of three or more outcomes, such ashigh quality blast, blast, and arrested.

In one embodiment, each series of images 2302 may be of an embryo thatwas previously frozen. This may facilitate in training of the rankingand/or categorization model 2306 for application to a group of embryosthat were previously frozen. Alternatively, each series of images 2302may be of an embryo that was not previously frozen. Alternatively, oneor more series of images 2302 may be of an embryo that was previouslyfrozen, and one or more different series of images 2302 may be of anembryo that was not previously frozen.

In one embodiment, the embryos to be ranked and/or categorized can beselected as a subset of a larger group of embryos. Referring to FIG. 8,the selection of the larger group of embryos can be received as an inputto the computing apparatus 804 via the input interface 808.Alternatively or in addition, an evaluation of an aspect of each embryosuch as morphology can be received as an input to the computingapparatus 804 via the input interface 808, and the selection of thelarger group of embryos can be performed based on the input.

In one embodiment, the embryos to be ranked and/or categorized can beselected as a subset of a larger group of embryos that can be determinedthrough cell activity tracking as described with reference to FIGS.16-22 and Examples 2 and 3, through classification approaches describedwith reference to FIGS. 1-15 and Example 1, through image similaritydetermination as described with reference to FIG. 19B and Example 3,and/or through combinations of the above approaches.

In one embodiment, the embryos can be ranked and/or categorized within apopulation of embryos that is selected based on an input to thecomputing apparatus 804 via the input interface 808. For example, theinput may indicate that the embryos to be ranked and/or categorized werepreviously frozen. In this case, the population of embryos may beselected to include only previously frozen embryos, or alternatively maybe selected to include both previously frozen embryos and embryos thatwere not previously frozen. In another example, the input may indicatethat the embryos to be ranked and/or categorized were not previouslyfrozen. In this case, the population of embryos may be selected toinclude only embryos that were not previously frozen, or alternativelymay be selected to include both previously frozen embryos and embryosthat were not previously frozen. In another example, the input mayspecify that the embryos are to be ranked and/or categorized within oneor more populations, such as embryos from a single patient, a singleclinic, a group of clinics, or an overall population.

FIG. 24 illustrates a non-limiting example of a display 2400 showing aresult of automated embryo ranking and embryo categorization, inaccordance with an embodiment of the invention. The display 2400 mayshow an identification 2402 of each of one or more wells 2404, and oneor more images 2406 of embryos. The display 2400 may also show a rankingscore 2408 for each embryo. In one embodiment, the ranking score 2408may be an intra-patient ranking score determined relative to otherembryos of the same patient. For example, the ranking score 2408 may benormalized, such as to the range [0, 1], such that, for example, thehighest ranked embryo of the patient can receive a score of 1.Alternatively or in addition, the ranking score 2408 may be determinedrelative to other embryos from a population, such as from a singleclinic, a group of clinics, or an overall population. The display 2400may also show a rank 2409, such as a rank within patient. Alternativelyor in addition, the display 2400 may also show a rank within apopulation, such as embryos from a single clinic, a group of clinics, oran overall population.

In one embodiment, the display 2400 may show an indicator 2410 of a rankin population. The rank in population may be determined relative toother embryos from a population, such as from a single clinic, a groupof clinics, or an overall population. The indicator 2410 may benormalized, such as to the range [0, 1]. Alternatively or in addition,the indicator 2410 may be expressed as a percentile indicator in therange [0%, 100%] such that, for example, the highest ranked embryo ofthe population can receive a score of 100%.

In one embodiment, the display 2400 may show category information 2412associated with each embryo. One example of a group of categories thatcan be shown as category information 2412 includes implant, freeze, anddiscard. Another example includes high quality blast, blast, andarrested.

Referring to FIG. 8, in some embodiments, the imaging device 802 can beconfigurable to acquire the images 2302, the training images for theranking and/or categorization module 2306, and/or the like. The imagingdevice 802 can also be configurable to acquire the images 2308. Thecomputing apparatus 804 is configured to perform the automated rankingand/or categorization of embryos described herein. In some embodiments,the display device 806 is at least configured to display one or moreimages of cells as acquired by the imaging device 802, and forpresenting ranking and/or categorization of embryos based on theautomated ranking and/or categorization described herein. In someembodiments, the input interface 808 is configured for a user to enterinput to the computing apparatus 804, as described herein.

Now referring to FIG. 9, in some embodiments, the memory 914 stores aset of executable programs (not shown) that are used to implement thecomputing apparatus 904 for automated embryo ranking and/orcategorization. Additionally or alternatively, the processor 912 can beused to implement the computing apparatus 904 for automated embryoranking and/or categorization. In such embodiments, the processor 912may include various combinations of the modules shown in FIG. 9,including but not limited to various combinations of one or more ofselection module 944, score determination module 948, ranking module950, categorization module 952, the display module 942, the image module920, the training module 934, the classification module 936, and thehypothesis selection module 926.

In one embodiment, the apparatus 904 for automated embryo rankingincludes the score determination module 948 configured to classify aplurality of images of each embryo included in a first plurality ofembryos to determine a score associated with a developmental potentialof each embryo included in the first plurality of embryos, and theranking module 950 configured to rank each embryo included in the firstplurality of embryos based on the score associated with each embryoincluded in the first plurality of embryos. The score determinationmodule 948 and the ranking module 950 are implemented in at least one ofa memory or a processing device.

In one embodiment, the apparatus 904 may be for automated embryocategorization in addition to or in the alternative to automated embryoranking. In this embodiment, the apparatus 904 includes the scoredetermination module 948 configured to classify a plurality of images ofeach embryo included in a first plurality of embryos to determine ascore associated with a developmental potential of each embryo includedin the first plurality of embryos, and the categorization module 952configured to associate each embryo included in the first plurality ofembryos with a corresponding category included in a plurality ofcategories based on the score associated with each embryo included inthe first plurality of embryos. The score determination module 948 andthe categorization module 952 are implemented in at least one of amemory or a processing device.

In one embodiment, the ranking module 950 is configured to rank eachembryo included in the first plurality of embryos within a secondplurality of embryos associated with a patient. Alternatively or inaddition, the ranking module 950 may be configured to rank each embryoincluded in the first plurality of embryos within a third plurality ofembryos associated with a population. The apparatus 904 for automatedembryo ranking may further include the display module 942 configured todisplay a ranking of each embryo included in the first plurality ofembryos within the second plurality of embryos associated with thepatient. Alternatively or in addition, the display module 942 may beconfigured to display a ranking of each embryo included in the firstplurality of embryos within the third plurality of embryos associatedwith the population.

In one embodiment, the ranking module 950 is configured to rank eachembryo included in the first plurality of embryos within a secondplurality of embryos. The apparatus 904 further comprises the selectionmodule 944 configured to select the second plurality of embryos based onan input to the apparatus 904.

In one embodiment, the categorization module 952 is configured tocategorize each embryo included in the first plurality of embryos withina second plurality of embryos. The apparatus 904 further comprises theselection module 944 configured to select the second plurality ofembryos based on an input to the apparatus 904.

In one embodiment of the apparatus 904 for automated embryo rankingand/or categorization, the input indicates that the first embryo waspreviously frozen, and in response to the input, the selection module944 is configured to select the second plurality of embryos such thateach embryo included in the second plurality of embryos was previouslyfrozen. Alternatively or in addition, the input indicates that the firstembryo was not previously frozen, and in response to the input, theselection module 944 is configured to select the second plurality ofembryos such that each embryo included in the second plurality ofembryos was not previously frozen. Alternatively or in addition, inresponse to the input, the selection module 944 is configured to selectthe second plurality of embryos such that the second plurality ofembryos includes both embryos that were previously frozen and embryosthat were not previously frozen.

In one embodiment, the apparatus 904 for automated embryo rankingfurther includes the selection module 944 configured to select one ormore of the first plurality of embryos for implantation into a femalehuman subject based on a ranking of each embryo included in the firstplurality of embryos based on the score. Alternatively or in addition,the selection module 944 may be configured to reject one or more of thefirst plurality of embryos for implantation into a female human subjectbased on a ranking of each embryo included in the first plurality ofembryos based on the score.

In one embodiment of the apparatus 904 for automated embryocategorization, each of the plurality of categories is associated with arange of scores. Each embryo may be associated with the correspondingcategory if the score associated with each embryo is in thecorresponding range of scores. In one embodiment, each embryo may beassociated with a corresponding clinic, and the range of scores may bebased on information associated with the corresponding clinic. Theinformation associated with the corresponding clinic may include but isnot limited to environmental parameters associated with thedevelopmental potential of each embryo. The environmental parameters mayinclude but are not limited to one or more of the following: media type;incubator temperature; and gas mix used in incubator.

In one embodiment, the apparatus 904 for automated embryo categorizationmay further include the display module 942 configured to display anassociation of each embryo with the corresponding category.

In one embodiment, the apparatus 904 for automated embryo categorizationfurther includes the selection module 944 configured to select one ormore of the first plurality of embryos for implantation into a femalehuman subject based on the corresponding category of each embryoincluded in the first plurality of embryos. Alternatively or inaddition, the selection module 944 may be configured to reject one ormore of the first plurality of embryos for implantation into a femalehuman subject based on the corresponding category of each embryoincluded in the first plurality of embryos.

In one embodiment of the apparatus 904 for automated embryo rankingand/or categorization, the score determination module 948 is configuredto determine the score based on an input to the apparatus 904, the inputbeing associated with at least one embryo included in the firstplurality of embryos. The input, for the at least one embryo, may beassociated with one or more of: an evaluation of morphology; an imagefeature having shape type; an image feature having texture type; animage feature having edge type; a machine learned image feature; abiological feature; a timing feature; a spatial-temporal feature;patient information; and clinic related information.

In one embodiment of the apparatus 904 for automated embryo rankingand/or categorization, the score may be substantially continuous. Thescore determination module 948 may be a classifier that is one of NaïveBayes, Adaboost, Support Vector Machine (SVM), Boosting, and RandomForrest.

In one embodiment of the apparatus 904 for automated embryo rankingand/or categorization, the score determination module 948 is configuredto determine the score based on environmental parameters associated withthe developmental potential of each embryo. The environmental parametersmay include one or more of the following: media type; incubatortemperature; and gas mix used in incubator.

In one embodiment of the apparatus 904 for automated embryo rankingand/or categorization, the score determination module 948 is configuredto determine the score for at least one embryo included in the firstplurality of embryos based on patient information associated with one ormore of a male parent of the at least one embryo, and a female parent ofthe at least one embryo. The patient information may include one or moreof the following: age; ethnicity; other demographic information;reproductive history; assisted reproductive history; for the maleparent, motility of sperm provided by the male parent and otherparameters associated with the sperm.

In one embodiment of the apparatus 904 for automated embryo rankingand/or categorization, the score determination module 948 is configuredto determine the score based on one or more of the following types ofimage features associated with one or more cells included in an embryoincluded in the first plurality of embryos: shape type, texture type,and edge type. Alternatively or in addition, the score determinationmodule 948 may be configured to determine the score based on a pluralityof image features including one or more machine learned image features.Alternatively or in addition, the score determination module 948 may beconfigured to determine the score based on one or more of the followingtypes of image features associated with a boundary of an embryo includedin the first plurality of embryos: cross-boundary intensity profile;cross-boundary texture profile; boundary curvature; and boundaryruffling.

In one embodiment of the apparatus 904 for automated embryo rankingand/or categorization, the score determination module 948 is configuredto determine the score based on one or more of the following factors:likelihood of each embryo reaching blastocyst stage; likelihood of eachembryo reaching blastocyst stage with a quality level; likelihood ofpregnancy resulting from each embryo; likelihood of implantationresulting from each embryo; likelihood of a genetic defect in eachembryo; likelihood of aneuploidy associated with each embryo; degree offragmentation associated with each embryo; and whether one or more ofthe first plurality of embryos will be frozen.

In one embodiment of the apparatus 904 for automated embryo rankingand/or categorization, the plurality of images includes atime-sequential series of images. Alternatively or in addition, theplurality of images may include: a first image of an embryo included inthe first plurality of embryos, the first image having a first depth offield; and a second image of the embryo, the second image having asecond depth of field different from the first depth of field.Alternatively or in addition, the plurality of images may include: afirst image of an embryo included in the first plurality of embryos, thefirst image having a first angular orientation relative to the embryo;and a second image of the embryo, the second image having a secondangular orientation relative to the embryo, the second angularorientation being different from the first angular orientation.Alternatively or in addition, the plurality of images may include afirst image of an embryo included in the first plurality of embryos, thefirst image having a first modality; and a second image of the embryo,the second image having a second modality different from the firstmodality. In one embodiment, each of the first modality and the secondmodality may be selected from the group consisting of brightfield anddarkfield.

In one embodiment, the apparatus 904 for automated embryo ranking and/orcategorization further comprises the training module 934 configured totrain the score determination module 948 based on at least one of labelinformation and feature information associated with one or more of aplurality of images of each embryo included in a second plurality ofembryos. The training module may be configured to extract the featureinformation, wherein the feature information includes one or more imagefeatures. The feature information may include one or more of: an imagefeature having shape type; an image feature having texture type; andimage feature having edge type; and a machine learned image feature.Each embryo included in the second plurality of embryos may have beenpreviously frozen. In one embodiment, the label information may includeone or more of: an indication of quality of the corresponding embryo;and an indication of one of a plurality of outcomes for development ofthe corresponding embryo. In one embodiment, the plurality of outcomesmay include one or more of the following pairs of outcomes: blast andarrested; implantation and no implantation; pregnancy and no pregnancy;and ploidy and aneuploidy. Alternatively or in addition, the pluralityof outcomes may include one or more sets of three or more outcomes. Theone or more sets of three or more outcomes may include the following setof three outcomes: high quality blast, blast, and arrested.

In one embodiment, the apparatus 904 for automated embryo ranking and/orcategorization is configured to determine a timing characteristicassociated with development of at least one embryo included in the firstplurality of embryos, wherein the score determination module 948 isconfigured to determine the score based on the timing characteristic.The apparatus 904 may further comprise the hypothesis selection module926 configured to select a hypothesis from a plurality of hypothesescharacterizing the at least one embryo, each of the plurality ofhypotheses including an inferred characteristic of the at least oneembryo based on a mapping of a representation of the at least one embryoto one or more boundary segments associated with the at least oneembryo, the apparatus 904 being configured to determine the timingcharacteristic based on the inferred characteristic associated with thehypothesis. Alternatively or in addition, the apparatus 904 may beconfigured to apply a plurality of classifiers to each of the pluralityof images to determine, for each image, a classification probabilityassociated with each classifier, wherein: each classifier is associatedwith a distinct first number of cells, and determines the classificationprobability for the each image based on a plurality of image features;and the classification probability indicates an estimated likelihoodthat the distinct first number of cells associated with each classifieris shown in each image, each of the plurality of images thereby having aplurality of the classification probabilities associated therewith. Theapparatus 904 may be further configured to classify each image asshowing a second number of cells based on the distinct first number ofcells associated with each classifier and the plurality ofclassification probabilities associated therewith.

In one embodiment, the apparatus 904 for automated embryo ranking and/orcategorization further comprises the image similarity determinationmodule 941 configured to determine image similarity between two or moreof the plurality of images, wherein the apparatus is configured todetermine the timing characteristic based on the image similarity.

In one embodiment, the apparatus 904 for automated embryo ranking and/orcategorization further comprises the selection module 944 configured toselect a subset of a second plurality of embryos as the first pluralityof embryos. The selection module 944 may be configured to select thesubset of the second plurality of embryos as the first plurality ofembryos based on an input to the apparatus. In one embodiment, the inputmay be associated with an evaluation of a morphology of thecorresponding embryo.

In one embodiment, the apparatus 904 for automated embryo ranking and/orcategorization further comprises the classification module 936configured to determine an estimated likelihood that a time-sequentialseries of images of each of the second plurality of embryos shows one ofa plurality of outcomes for development of each of the second pluralityof embryos. The selection module 944 may be configured to select thesubset of the second plurality of embryos as the first plurality ofembryos based on the estimated likelihood associated with each of thesecond plurality of embryos. In one embodiment, the plurality ofoutcomes may include one or more of the following pairs of outcomes:blast and arrested; implantation and no implantation; pregnancy and nopregnancy; and ploidy and aneuploidy. Alternatively or in addition, theplurality of outcomes may include one or more sets of three or moreoutcomes. The one or more sets of three or more outcomes may include thefollowing set of three outcomes: high quality blast, blast, andarrested.

FIG. 25 illustrates a method for embryo ranking and/or embryocategorization, in accordance with an embodiment of the invention. Themethod includes applying one or more of a classifier and a neuralnetwork to a plurality of images of each embryo included in a firstplurality of embryos to determine a score associated with adevelopmental potential of each embryo included in the first pluralityof embryos (block 2530). The method also includes ranking each embryoincluded in the first plurality of embryos based on the score associatedwith each embryo included in the first plurality of embryos (block2540). Alternatively or in addition, the method may include associatingeach embryo included in the first plurality of embryos with acorresponding category included in a plurality of categories based onthe score associated with each embryo included in the first plurality ofembryos (block 2550).

In one embodiment, the method may include selecting a subset of embryosfor ranking and/or categorization (block 2510). The method may includetraining the one or more of the classifier and the neural network basedon at least one of label information and feature information associatedwith images of embryos (block 2520).

In one embodiment, the method includes ranking each embryo included inthe first plurality of embryos within a second plurality of embryosassociated with a patient. Alternatively or in addition, the method maybe include ranking each embryo included in the first plurality ofembryos within a third plurality of embryos associated with apopulation. The method may further include displaying a ranking of eachembryo included in the first plurality of embryos within the secondplurality of embryos associated with the patient. Alternatively or inaddition, the method may include displaying a ranking of each embryoincluded in the first plurality of embryos within the third plurality ofembryos associated with the population.

In one embodiment, the method includes receiving an input associatedwith a first embryo included in the first plurality of embryos, wherethe ranking each embryo included in the first plurality of embryosincludes ranking the first embryo within a second plurality of embryos.The method may further include selecting the second plurality of embryosbased on the input.

In one embodiment, the method includes receiving an input associatedwith a first embryo included in the first plurality of embryos, wherethe associating each embryo included in the first plurality of embryoswith the corresponding category includes categorizing each embryo withina second plurality of embryos. The method may further include selectingthe second plurality of embryos based on the input.

In one embodiment of the method for automated embryo ranking and/or themethod for automated embryo categorization, the input indicates that thefirst embryo was previously frozen, and in response to the input, thesecond plurality of embryos are selected such that each embryo includedin the second plurality of embryos was previously frozen. Alternativelyor in addition, the input indicates that the first embryo was notpreviously frozen, and in response to the input, the second plurality ofembryos are selected such that each embryo included in the secondplurality of embryos was not previously frozen. Alternatively or inaddition, in response to the input, the second plurality of embryos areselected such that the second plurality of embryos includes both embryosthat were previously frozen and embryos that were not previously frozen.

In one embodiment, the method for automated embryo ranking furtherincludes selecting one or more of the first plurality of embryos forimplantation into a female human subject based on a ranking of eachembryo included in the first plurality of embryos based on the score.Alternatively or in addition, the method may include rejecting one ormore of the first plurality of embryos for implantation into a femalehuman subject based on a ranking of each embryo included in the firstplurality of embryos based on the score.

In one embodiment of the method for automated embryo categorization,each of the plurality of categories is associated with a range ofscores. Each embryo may be associated with the corresponding category ifthe score associated with each embryo is in the corresponding range ofscores. In one embodiment, each embryo may be associated with acorresponding clinic, and the range of scores may be based oninformation associated with the corresponding clinic. The informationassociated with the corresponding clinic may include but is not limitedto environmental parameters associated with the developmental potentialof each embryo. The environmental parameters may include but are notlimited to one or more of the following: media type; incubatortemperature; and gas mix used in incubator.

In one embodiment, the method for automated embryo categorization mayfurther include displaying an association of each embryo with thecorresponding category.

In one embodiment, the method for automated embryo categorizationfurther includes selecting one or more of the first plurality of embryosfor implantation into a female human subject based on the correspondingcategory of each embryo included in the first plurality of embryos.Alternatively or in addition, the method may include rejecting one ormore of the first plurality of embryos for implantation into a femalehuman subject based on the corresponding category of each embryoincluded in the first plurality of embryos.

In one embodiment of the method for automated embryo ranking and/or themethod for automated embryo categorization the score is substantiallycontinuous, and is determined by the classifier. The classifier may beone of Naïve Bayes, Adaboost, Support Vector Machine (SVM), Boosting,Random Forrest, or any other suitable classifier.

In one embodiment of the method for automated embryo ranking and/or themethod for automated embryo categorization, the method includesreceiving an input associated with at least one embryo included in thefirst plurality of embryos, where the applying the one or more of theclassifier and the neural network includes determining the score basedon the input. The input, for the at least one embryo, may be associatedwith one or more of: an evaluation of morphology; an image featurehaving shape type; an image feature having texture type; an imagefeature having edge type; a machine learned image feature; a biologicalfeature; a timing feature; a spatial-temporal feature; patientinformation; and clinic related information.

In one embodiment of the method for automated embryo ranking and/or themethod for automated embryo categorization, the applying the one or moreof the classifier and the neural network includes determining the scorebased on environmental parameters associated with the developmentalpotential of each embryo. The environmental parameters may include oneor more of the following: media type; incubator temperature; and gas mixused in incubator.

In one embodiment of the method for automated embryo ranking and/or themethod for automated embryo categorization, the applying the one or moreof the classifier and the neural network includes determining the scorefor at least one embryo included in the first plurality of embryos basedon patient information associated with one or more of a male parent ofthe at least one embryo, and a female parent of the at least one embryo.The patient information may include one or more of the following: age;ethnicity; other demographic information; reproductive history; assistedreproductive history; for the male parent, motility of sperm provided bythe male parent and other parameters associated with the sperm.

In one embodiment of the method for automated embryo ranking and/or themethod for automated embryo categorization, the applying the one or moreof the classifier and the neural network is based on one or more of thefollowing types of image features associated with one or more cellsincluded in an embryo included in the first plurality of embryos: shapetype, texture type, and edge type. Alternatively or in addition, theapplying the one or more of the classifier and the neural network isbased on a plurality of image features including one or more machinelearned image features. Alternatively or in addition, the applying theone or more of the classifier and the neural network is based on one ormore of the following types of image features associated with a boundaryof an embryo included in the first plurality of embryos: cross-boundaryintensity profile; cross-boundary texture profile; boundary curvature;and boundary ruffling.

In one embodiment of the method for automated embryo ranking and/or themethod for automated embryo categorization, the applying the one or moreof the classifier and the neural network includes determining the scorebased on one or more of the following factors: likelihood of each embryoreaching blastocyst stage; likelihood of each embryo reaching blastocyststage with a quality level; likelihood of pregnancy resulting from eachembryo; likelihood of implantation resulting from each embryo;likelihood of a genetic defect in each embryo; likelihood of aneuploidyassociated with each embryo; degree of fragmentation associated witheach embryo; whether one or more of the first plurality of embryos willbe frozen; whether one or more of the embryos resulted fromintra-cytoplasmic sperm injection; whether the one or more of the firstplurality of embryos resulted from traditional in-vitro fertilization;and medications used in connection with in vitro fertilization resultingin one or more of the embryos.

In one embodiment of the method for automated embryo ranking and/or themethod for automated embryo categorization, the plurality of imagesincludes a time-sequential series of images. Alternatively or inaddition, the plurality of images may include: a first image of anembryo included in the first plurality of embryos, the first imagehaving a first depth of field; and a second image of the embryo, thesecond image having a second depth of field different from the firstdepth of field. Alternatively or in addition, the plurality of imagesmay include: a first image of an embryo included in the first pluralityof embryos, the first image having a first angular orientation relativeto the embryo; and a second image of the embryo, the second image havinga second angular orientation relative to the embryo, the second angularorientation being different from the first angular orientation.Alternatively or in addition, the plurality of images may include: afirst image of an embryo included in the first plurality of embryos, thefirst image having a first modality; and a second image of the embryo,the second image having a second modality different from the firstmodality. In one embodiment, each of the first modality and the secondmodality may be selected from the group consisting of brightfield anddarkfield.

In one embodiment, the method for automated embryo ranking and/or themethod for automated embryo categorization further includes training theone or more of the classifier and the neural network based on at leastone of label information and feature information associated with one ormore of a plurality of images of each embryo included in a secondplurality of embryos. The training the one or more of the classifier andthe neural network may include extracting the feature information,wherein the feature information includes one or more image features. Thefeature information may include one or more of: an image feature havingshape type; an image feature having texture type; and image featurehaving edge type; and a machine learned image feature. Each embryoincluded in the second plurality of embryos may have been previouslyfrozen. In one embodiment, the label information may include one or moreof: an indication of quality of the corresponding embryo; and anindication of one of a plurality of outcomes for development of thecorresponding embryo. In one embodiment, the plurality of outcomes mayinclude one or more of the following pairs of outcomes: blast andarrested; implantation and no implantation; pregnancy and no pregnancy;and ploidy and aneuploidy. Alternatively or in addition, the pluralityof outcomes may include one or more sets of three or more outcomes. Theone or more sets of three or more outcomes may include the following setof three outcomes: high quality blast, blast, and arrested.

In one embodiment, the method for automated embryo ranking and/or themethod for automated embryo categorization further includes determininga timing characteristic associated with development of at least oneembryo included in the first plurality of embryos, wherein the applyingthe one or more of the classifier and the neural network includesdetermining the score based on the timing characteristic. Thedetermining the timing characteristic may include selecting a hypothesisfrom a plurality of hypotheses characterizing the at least one embryo,each of the plurality of hypotheses including an inferred characteristicof the at least one embryo based on a mapping of a representation of theat least one embryo to one or more boundary segments associated with theat least one embryo. The determining the timing characteristic may alsoinclude determining the timing characteristic based on the inferredcharacteristic associated with the hypothesis. The determining thetiming characteristic may include selecting a hypothesis from aplurality of hypotheses characterizing the at least one embryo, each ofthe plurality of hypotheses including an inferred characteristic of theat least one embryo based on geometric features of the at least oneembryo.

Alternatively or in addition, in the method for automated embryo rankingand/or the method for automated embryo categorization, the classifiermay be a first classifier. The method may further include applying aplurality of second classifiers to each of the plurality of images todetermine, for each image, a classification probability associated witheach second classifier, wherein: each second classifier is associatedwith a distinct first number of cells, and determines the classificationprobability for the each image based on a plurality of image features;and the classification probability indicates an estimated likelihoodthat the distinct first number of cells associated with each secondclassifier is shown in each image, each of the plurality of imagesthereby having a plurality of the classification probabilitiesassociated therewith. The method may further include classifying eachimage as showing a second number of cells based on the distinct firstnumber of cells associated with each second classifier and the pluralityof classification probabilities associated therewith.

In one embodiment of the method for automated embryo ranking and/or themethod for automated embryo categorization, the determining the timingcharacteristic may include determining image similarity between two ormore of the plurality of images.

In one embodiment, the method for automated embryo ranking and/or themethod for automated embryo categorization includes receiving an inputspecifying the first plurality of embryos.

In one embodiment, the method for automated embryo ranking and/or themethod for automated embryo categorization includes receiving an inputassociated with each embryo included in a second plurality of embryos,and selecting a subset of the second plurality of embryos as the firstplurality of embryos based on the input. In one embodiment, the input,for each embryo, may be associated with one or more of: an evaluation ofmorphology; an image feature having shape type; an image feature havingtexture type; an image feature having edge type; a machine learned imagefeature; a biological feature; a timing feature; a spatial-temporalfeature; patient information; and clinic related information.

In one embodiment of the method for automated embryo ranking and/or themethod for automated embryo categorization, the selecting the subsetincludes determining an estimated likelihood that a time-sequentialseries of images of each of the second plurality of embryos shows one ofa plurality of outcomes for development of each of the second pluralityof embryos, the plurality of outcomes being associated with theclassifier. The method may further include selecting the subset of thesecond plurality of embryos as the first plurality of embryos based onthe estimated likelihood associated with each of the second plurality ofembryos. In one embodiment, the plurality of outcomes may include one ormore of the following pairs of outcomes: blast and arrested;implantation and no implantation; pregnancy and no pregnancy; and ploidyand aneuploidy. Alternatively or in addition, the plurality of outcomesmay include one or more sets of three or more outcomes. The one or moresets of three or more outcomes may include the following set of threeoutcomes: high quality blast, blast, and arrested.

An embodiment of the invention relates to a computer storage productwith a computer-readable medium having computer code thereon forperforming various computer-implemented operations. The term“computer-readable medium” is used herein to include any medium that iscapable of storing or encoding a sequence of instructions or computercodes for performing the operations described herein. The media andcomputer code may be those specially designed and constructed for thepurposes of the invention, or they may be of the kind well known andavailable to those having skill in the computer software arts. Examplesof computer-readable media include, but are not limited to: magneticmedia such as hard disks, floppy disks, and magnetic tape; optical mediasuch as CD-ROMs and holographic devices; magneto-optical media such asfloptical disks; and hardware devices that are specially configured tostore and execute program code, such as application-specific integratedcircuits (“ASICs”), programmable logic devices (“PLDs”), and ROM and RAMdevices. Examples of computer code include machine code, such asproduced by a compiler, and files containing higher-level code that areexecuted by a computer using an interpreter or a compiler. For example,an embodiment of the invention may be implemented using Java, C++, orother object-oriented programming language and development tools.Additional examples of computer code include encrypted code andcompressed code. Moreover, an embodiment of the invention may bedownloaded as a computer program product, which may be transferred froma remote computer (e.g., a server computer) to a requesting computer(e.g., a client computer or a different server computer) via atransmission channel. Another embodiment of the invention may beimplemented in hardwired circuitry in place of, or in combination with,machine-executable software instructions.

An embodiment of the invention can be implemented in hardware, such as afield programmable gate array (FPGA) or ASIC. The FPGA/ASIC may beconfigured by and may provide output to input/output devices.

The preceding merely illustrates the principles of the invention. It isappreciated that those skilled in the art may be able to devise variousarrangements which, although not explicitly described or shown herein,embody the principles of the invention and are included within itsspirit and scope. The illustrations may not necessarily be drawn toscale, and manufacturing tolerances may result in departure from theartistic renditions herein. There may be other embodiments of thepresent invention which are not specifically illustrated. Thus, thespecification and the drawings are to be regarded as illustrative ratherthan restrictive. Additionally, the drawings illustrating theembodiments of the present invention may focus on certain majorcharacteristic features for clarity. Furthermore, all examples andconditional language recited herein are principally intended to aid thereader in understanding the principles of the invention and the conceptscontributed by the inventors to furthering the art, and are to beconstrued as being without limitation to such specifically recitedexamples and conditions. Moreover, all statements herein recitingprinciples, aspects, and embodiments of the invention as well asspecific examples thereof, are intended to encompass both structural andfunctional equivalents thereof. Additionally, it is intended that suchequivalents include both currently known equivalents and equivalentsdeveloped in the future, i.e., any elements developed that perform thesame function, regardless of structure. The scope of the presentinvention, therefore, is not intended to be limited to the exemplaryembodiments shown and described herein. Rather, the scope and spirit ofthe present invention is embodied by the appended claims. In addition,while the methods disclosed herein have been described with reference toparticular operations performed in a particular order, it will beunderstood that these operations may be combined, sub-divided, orre-ordered to form an equivalent method without departing from theteachings of the invention. Accordingly, unless specifically indicatedherein, the order and grouping of the operations are not limitations ofthe invention. All references cited herein are incorporated by referencein their entireties.

What is claimed is:
 1. A method for automated cell classification,comprising: acquiring a series of time-sequential images of at least onehuman embryo comprising one or more cells contained in a multi-wellculture dish comprising a plurality of microwells with a camera of atleast one time-lapse microscope; extracting image-based features fromthe series of time-sequential images; applying a model to theimage-based features; applying a plurality of first classifiers to eachof the series of time-sequential images of at least one human embryo todetermine, for each image, a first classification probability associatedwith each first classifier, wherein: each first classifier is associatedwith a distinct first number of cells, and determines the firstclassification probability for the each image based on a plurality ofcell features including one or more machine learned cell features; andthe first classification probability indicates a first estimatedlikelihood that the distinct first number of cells associated with theeach first classifier is shown in the each image, the each of the seriesof time-sequential images thereby having a plurality of the firstclassification probabilities associated therewith; and classifying eachimage as showing a second number of cells based on the distinct firstnumber of cells associated with the each first classifier and theplurality of first classification probabilities associated therewith,wherein the classification of each image as showing a second number ofcells comprises one of the extracted image-based features; generating ananeuploidy prediction for the at least one human embryo with the modeland the image-based features.
 2. The method of claim 1, wherein theaneuploidy prediction comprises a categorization of the human embryointo one of several groups based on a score.
 3. The method of claim 1,further comprising providing a recommendation based on the aneuploidyprediction.
 4. The method of claim 3, wherein the recommendation isprovided via a display.
 5. The method of claim 3, wherein therecommendation comprises one of: a selection of the human embryo forimplantation; or a rejection of the human embryo.
 6. The method of claim5, wherein the model comprises at least one of: a classifier; or aneural network.
 7. The method of claim 6, wherein the model is generatedvia machine learning.
 8. A system for non-invasively predictinganeuploidy of a human embryo from a donor with an imaging system, thesystem comprising: a stage configured to receive a multi-well culturedish comprising a plurality of micro-wells containing a samplecomprising at least one human embryo; a time-lapse microscope includinga camera, a memory device, and a processor configured to acquiretime-sequential images of the multi-well culture dish on the stage,wherein the time-lapse microscope is configured to: extract image-basedfeatures from the series of time-sequential images and apply a model tothe image-based features; apply a plurality of first classifiers to eachof the series of time-sequential images of at least one human embryo todetermine, for each image, a first classification probability associatedwith each first classifier, wherein: each first classifier is associatedwith a distinct first number of cells, and determines the firstclassification probability for the each image based on a plurality ofcell features including one or more machine learned cell features; andthe first classification probability indicates a first estimatedlikelihood that the distinct first number of cells associated with theeach first classifier is shown in the each image, the each of theplurality of images thereby having a plurality of the firstclassification probabilities associated therewith; and classify eachimage as showing a second number of cells based on the distinct firstnumber of cells associated with the each first classifier and theplurality of first classification probabilities associated therewith,wherein the classification of each image as showing a second number ofcells comprises one of the extracted image-based features; generate ananeuploidy prediction for the at least one human embryo with the modeland the image-based features, wherein generating the aneuploidyprediction comprises generating a score associated with the developmentpotential of the human embryo.
 9. The system of claim 8, wherein thescore associated with the developmental potential of the human embryo isrelative with respect to a plurality of other human embryos from thedonor.
 10. The system of claim 9, wherein the aneuploidy predictioncomprises a categorization of the human embryo into one of severalgroups based on the score.
 11. The system of claim 8, wherein thetime-lapse microscope is further configured to generate a recommendationbased on the aneuploidy prediction.
 12. The system of claim 11, furthercomprising a display configured to provide a recommendation based on theaneuploidy prediction.
 13. The system of claim 12, wherein therecommendation comprises one of: a selection of the human embryo forimplantation; or a rejection of the human embryo.
 14. The system ofclaim 13, wherein the model comprises at least one of: a classifier; ora neural network.